Syncing projects with Google Drive - Part 4 (Handling Subfolders and Duplicates)

I'm back. We're going to continue building the DriveSyncer project. This is probably part four, I guess. I don't really keep track of those things. I just record. I just build features, record them, shove them to YouTube. Anyway, that was a good intro, right? Anyway, in the previous video, we built a couple of new features. We uploaded, not uploaded, we created this upload folder function, which is going to upload our project folder and all of the files inside it to Google Drive. The second thing we did is we created this base folder function over here that's going to create a base folder where we store all of our projects inside of it, right? So pretty good progress for one video. In this video, I want to create one or two new features, depending on how much time we have. I just started recording. I'm going to try and keep this to like 15-20 minutes of recording. So we'll see how many features we can build in that time frame. First of all, I want to be able to delete any previous folders. So let me show you a problem over here. Basically, in my Google Drive, we have this folder that I just think, right? But you will recall that I told you in the previous video that Google Drive does not care about folder names as unique, right? It just going to duplicate the entire folder if you try to upload the same folder twice. And we'll see that in just a second. In fact, we can already see it over here. You can see instead of updating this folder, it just created a new folder, a duplicate folder with the exact same file contents, right? We don't want that, obviously. And there's an easy way to fix this issue, and there's a very complex way to fix this issue. I'm going to go with the easy way, which is to just delete the folder and then re-upload it. The complex way is to basically track the file changes for every single file, track the, I guess, the MD5 or the SHA-256 file checksums, and just make sure that there were no changes in the file. If there were changes in a specific file, like for instance, if this main.go file was unchanged, but this sync.go file was changed, I added some new code over here, then what we should do in that case is leave the main file unchanged and just leave it on Google Drive, whereas we would upload this sync file because this is the one that changed. That's a much more complicated approach to solving this problem. Why? Because you have to keep track of every single file that changes. You have to keep track of every single checksum for every single file, and it just gets more complicated, right? We're not going to do that. What I'm going to do in this project is I'm going to literally just delete the previous folder and then just re-upload it. So to do that, let's go down here and create a function. We'll call this `deletePreviousFolder`. This will take the drive, obviously. It'll also take the base folder ID, which will be a string. You'll see why we need this in just a second. And it will take the folder as well, which is the folder we're trying to delete. And the only thing it can return is an error. So if there's an error in trying to delete this previous folder, we'll return that. Now, in order to delete the previous folder, we want to first find it, right? And quick hint, we did something similar up here in the create base folder function, right? You basically want to create a query like this, and you want to use that to find the previous folder. So let's create a query over here. Let's say `fmt`. We say MIME type, wrap it in single quotes, add a string over here, and print the folder MIME type over here. We'll say that this is a folder, and the name of this folder wrapped in single quotes is going to be `folder.name`. And again, wrap in single quotes a string over here in parents. And here we'll say the base folder ID. So we're saying that the base folder is a parent of this folder, this project folder, because every single project that we upload is going to be inside this base folder, right? So we're basically just checking that over here, right? And trash equals false, right? So quick review. All I'm saying over here is check the MIME type to make sure this is a folder, check the folder name to make sure this is the one we're looking for, check to make sure it is inside the base folder, which is where we store all of our projects, and check to make sure that it's not trashed. It's not inside the recycle bin. It is an actual folder. Right? That's the query. Now let's run this. Let's say `drive.api.files.list.q`, pass in the query, `.fields`, and here we can say `files(id, name)`. And one more function, we can say page size equals one. So basically, limit the search results to only one search result because they should be unique, right? Every single project should have a unique folder name, right? And that's all we need to do. Let's run `do`, grab the response and an error over here. Let's handle the error as well. If there is an error, let's just return it over here. Now let's say `log.Printf`, attempting to delete previous folder for project, and then print the folder name over here with a new line. So pass in the folder name. That'll be `folder.name`. And then we need to add a couple of checks over here. So first of all, let's say something like `if len(r.Files) == 0`. So basically, if we were not able to find this folder, if this folder does not exist, what does that mean? It means that this folder does not exist, and we can just upload it new, right? So let's say something like, "Previous folder does not exist, uploading new," and just return nil, right? This is what's going to happen if you're uploading a project for the first time, right? It's not going to be on Google Drive by default, right? If you're uploading a new project, it not going to be on Google Drive in the first place. So this conditional will run, you return nil, and you just continue uploading the folder as expected, right? Else if `len(r.Files) == 1`, which means that this folder does exist already, this is where we actually need to delete it. So I'm going to say `log.Printf("Deleting previous folder")`, print the folder name with a new line as well, pass the folder name over here, and then I'm not going to return from the function over here. So a final check over here, else if `len(r.Files)` is, let's say, greater than one. So basically, there's duplicate folders with the same name in this folder, right? There's duplicate folders, duplicate project folders inside the base folder with this exact same name, right, that match this whole query, right? If that's the case, then we should say something like `log.Printf` or not `log.Printf`, but like `return errors.New`. We'll basically create a new error over here, and we'll say, "Multiple folder matches returned from API. Cannot delete," right? Because if you find multiple folders with this exact same query, which one do you delete, right? Which one is the old folder? Which one is the new? Which one are you supposed to delete? Which one are you supposed to keep? You have no idea, right? So we just skip that, return an error. Adding it. Um, there are three conditionals here. In this one, we return. In this one, we return. But in the second one, we were supposed to delete the previous folder. We don't return, and the reason is because down here, I'm going to delete that folder. This is going to be pretty simple. We'll just say `drive.api.files.Delete`, pass in the folder ID, which will be `r.Files[0].Id` (the first element in there), and then one. Do to actually execute this, we'll get a response and stuff, or actually, we won't get a response. We'll get an error over here. Not a response, so let's just say `err =` this. Since `err` is already declared up here, we don't need a colon. Remove that. If there's an error, let's return it. And if there's no error, let's return nil. So, with all of that done, error strings should not end with punctuation or new lines. Okay, let's delete this full stop over here, and that should be fine. Perfect. Now back up here in the upload folder function. Let's actually use this. So right after we create the base folder, we can say something like, `if err = deletePreviousFolder`. Why can't I get IntelliSense to work in these scenarios? Error does not equal to nil. Let me just do it this way, guys. If delete previous folder, pass in the drive. Now it works. Pass in the base folder ID and pass in the folder as well. So all I'm saying is try to delete the previous folder. Grab the error value that you get from this function, right? Because this is going to return an error. Grab that error value, put it in this `err` variable, and if that error is nil, or if it's not nil, if there is an error in trying to delete that folder, let's say `log.Printf`, let's say, "Failed to delete previous folder. You may see duplicate in your upload." And then we can print the error over here with a new line. So print the error. And you might wonder why I'm not using `log.Fatalf` over here to exit the program, and why I'm only using `log.Printf` and allowing the program to still continue to run. Well, the reason is, since Google Drive does allow duplicates, I kind of just, what I'm going to do is let the upload continue. We're not going to crash the program since this is not a critical issue. We can still proceed with the upload after this, but we do need to inform the user that this is going to occur. So that's what the log here is for. Now, hopefully I did everything correctly. I'm going to delete everything in my Google Drive so we start fresh. We're going to first of all just upload everything once more, and this should work perfectly fine. As you can see, attempting to delete previous folder. Previous folder does not exist. It's basically going to try to delete it, and if it doesn't exist already, it's just going to upload a new folder, which is what it's doing right now. Okay, almost finished. There we go, and here's the folder, and here are all the files. Perfect. This is what we had before as well, right? So what's changed? Well, if I run the whole thing again, you know that previously it was going to create a duplicate folder over here, right? But this time, this time it's going to, where's my, there it is, it's going to attempt to delete the previous folder. And you can see over here, deleting previous folder DriveSyncorVideo. It deleted the previous folder, and it uploading all of these new files to the new folder over here. And you can see all of them exist. All of them were uploaded successfully. Awesome. So this functionality works. We can now delete the previous folders and upload all of the new files successfully. The second thing I want to implement, and hopefully the video isn't getting too long. Okay, 14 minutes. We still have time. We still have plenty of time. The second thing that I want to implement is subfolders. So you can see over here, um, where is, for instance, this test.txt file. You see over here? Yeah, you can see I have these subfolders over here, and inside this subfolder is the test.txt file. But on Google Drive, you don't see any subfolder, right? This test.txt file just got uploaded to the root folder over here, right? To the root project folder, instead of a subfolder. So we need to implement the subfolder functionality. Before I do that, I want to clean up this code a little bit. I want to say something like, um, let's grab all of this and try to put it inside a different function. Let's say `uploadFile`, and there we go. So this is going to require a bunch of arguments. Let's try to add all of them over here. Path as well, I guess, which will be a string. It's going to return an error. Let's add that over here as well. It's going to also need the `d` variable, which is `fs.DirEntry`, the directory over here, and the project folder as well. So let's put that over here as well. This little, this will go, what am I saying? This is going to be a `drive.File` pointer, and that's all we need to do over here, right? Awesome. Now we can just run this. So over here in the `filepath.WalkDir` function, you can see if the directory, or if the object that we're looking at over here is a directory, then we return nil. But if it's a file, then we should run `uploadFile`. We should pass in the arguments, the drive, the path, the `d` entry, and the project folder over here. And this should still work, right? If it doesn't work, then we grab the error over here and we return the error as such, or you can even simplify this to something like `return` this, and that'll work as well. Let's do one quick upload just to make sure that this is still working and that we didn't break something over here. Okay, it's still uploading files. Perfect. So this works. Now we can upload files through this function, but now we also want to upload folders over here, right? Instead of `return nil`, this is going to be a bit more complicated, but let's try to work through this. Let's say create subfolder. I'm going to move this functionality to its own subfunction over here, right? This will take the drive. It'll take the parent, or actually the parent ID, which will just be a string, and it will take the fs.DirEntry as well. And it can return a Google Drive file along with an error message. Right now, the subfolder that we create over here, which will be drive.File. What was that file? There we go. The name will just be d.Name. The MIME type, again, as you know, every single Google Drive folder must have this MIME type over here, right? And one more thing, it must also have a parent, all right? And the parent will be a list of strings, and we'll pass in the parent ID over here. This is how we create a tree of directories, right? We'll basically create subfolders within subfolders by using this parent ID over here. And now we can just say something like response and error equals drive.API.Files.Create. Pass in the sub variable over here as a reference. Run this. Do this should be very simple because we've created multiple folders like this already, right? So you should have a very good idea of what I'm doing over here. Make sure to handle the error as well. We can say something like return nil and error. Otherwise, let's return r and nil over here, and that should be fine. And voila, now we can create subfolders, but we're not done yet. We also need to figure out the exact parents for every single one of these subfolders, right? So let's create the functionality to do that. First of all, let's create a map over here of subfolders. This will be a map of string of string. And the first one I'm going to add over here is the base folder as such. There we go. And let's fix this name as well. Basically, this will be a map of subfolders. We're going to put all of the subfolders in this project folder in this map. The key will be the folder name, and the ID over here, the value, will be the folder ID on Google Drive, right? So here where this is useful, we're basically going to store all of the subfolders that we create inside of this map. And then every time we want to upload a file within that subfolder, we'll just give the folder, the subfolder name over here, and we'll get the subfolder ID. And then we can use that to put that file inside that parent folder in Google Drive, right? That is why we use a map over here. If this doesn't make sense right now, it will once we implement this. Now inside this, um, d.IsDir, we're going to say subName. This will be the subfolder name, and we'll use filepath.Base and pass in the path over here. This will return the last element of the path, all right? This will be the folder name, the name of the subfolder, right? So let me walk you through this. You can see that this check over here is going to make sure that this is a directory, right? So if I have a path like something folder slash subfolder slash subfolder, right? filepath.Base will give me the... Where's the hover? Why can't I hover over this? Okay, anyway, yeah, there we go. filepath.Base will return the last element of the path, right? In this case, the last element of the path is this subfolder right here, right? This subfolder. So this will return the subfolder name. And then we need the parent name, all right? So which parent folder is the parent of this subfolder, right? We want to grab that as well so we can place this subfolder inside the appropriate directory, right? For that, what we're going to do is first of all run filepath.Dir. This will return all but the last element of the path, and then we'll pass in the path over here. And then we'll say filepath.Base on this as well. This will give us the parent name. So again, let me show you an example over here inside of a comment. Let me actually zoom in as well. I don't know if you guys can see the code as well as I can, but let me zoom in. If I have a path like, let's say, project slash folder slash subfolder, right? Never mind, let's just keep it this simple. So in a path like this, the subName is going to be the subfolder over here, and the parentName is going to be folder over here, right? We want to make sure that we upload this subfolder inside of this folder over here, right? This parent. We don't want to, like, if this is the local path on our machine, we don't want to have something like project slash subfolder on our Google Drive, right? We want to make sure that we upload it to the appropriate folder like this, right? That's what I'm trying to do over here. So let's zoom out again. The subName is the name of the subfolder. The parentName is the name of the parent folder that houses this subfolder, right? And we need both if we want to upload this properly. Now we can just run a conditional like if, okay, subfolders. Now we can use this map, pass in the subName, and run if not okay, if this doesn't exist over there. Then we can say something like parentID and pOK, which is parentOK, and run the subfolders again and pass in the parentName. So basically, if this subfolder doesn't exist already, then we need to create it, right? So we'll create it inside of this conditional, inside of this block of code over here, right? And before we create that subfolder, we want to grab the ID of the parent, right? Which we will do from this subfolders map, all right? So let's say var parent string. This will be the actual ID. Let's say that if it exists, right? If the parent ID exists, that's what this pOK variable is going to tell us, then we'll say parent equals the parent ID, right? Otherwise, if this parent doesn't exist, right? If this parent doesn't exist, then the parent will just be the base folder and its ID. And now we can finally create that folder over here by running the createSubfolder function, passing the drive, passing The parent passing `d`, which is the drive, the directory entry over here. This will return two things. It will return the subfolder and an error over here. Make sure to handle the error as well. Let's just say `return error`, and let's say `log.Println("Failed to create subfolder")`. Instead of `Println`, let's use `Printf`. I like using `Printf` there we go. Print the subfolder name over here and print the error after that. Add a new line. `sub.Name` and `error`. There we go. Now, assuming this worked, let's add this to the map of subfolders we have up here, right? We'll say `subfolders[subName]` because the key is the name of the folder, and the value in this map is going to be the ID of that folder on Google Drive, right? And then we can add a bunch of logs again over here. We can say something like, `log.Printf("Created subfolder")`. Print the subfolder name over here. In parent, we can print the parent as well, and I think that all we need to do right now. We want to make sure that we upload every single file to the appropriate subfolder as well. So this will take a bit more code. And by the way, before I do that, let me explain all of this as well real quick. We first of all make sure that we, or actually, we first of all make sure that we are inside of a directory, right? If we are inside of the directory, then this `subName` will be the name of the subfolder that we're trying to create, and this `parentName` will be the parent folder of that subfolder, right? So we make sure that we create the subfolders inside of the appropriate folders, right? Everything else is pretty straightforward. If it already exists, then we do nothing. But if it doesn't exist, if this subfolder doesn't exist, then we create the subfolder over here, right? And we make sure to upload it to the appropriate parent folder by using this block of code over here, right? And then finally, we just make sure to add that new subfolder to the subfolders map so we can use it in uploading new files and folders to that subfolder, right? Super simple. Now, in this else block where we're uploading files, let's say `parentName` equals `filepath.Base(filepath.Dir(path))`, and then pass in the path over here. And then let's say, if `parentID, ok := subfolders[parentName]`, pass in the parent name over here, and then make sure that we check for `ok` over here. If this works, then we should just say `return uploadFile` over here. Else, or by the way, we should, instead of using project folder over here for the parent, we should probably pass in `parentID`. And that'll be... Wait a second, this doesn't accept parents. Okay, we have a problem over here. In this `uploadFile` function, instead of using the project folder, this should be renamed to `parentID`, and it should take a string, not a `drive.File`, so we can use that over here as such. There we go. That's all we need to do, and then just make sure to pass that in over here, right? Else, if we can't find the parent for this folder, for this file, let's just return an error. And before we return an error, let's print the error as well. Let's say `log.Printf("Error")`, and the error over here will just be, I don't know, "Failed to find parent folder". Print the folder name over here for file and print the file over here, and just say upload failed. Pass that in over here and in `errors.New` as well. And instead of this empty string, let's use `fmt.Sprintf` so I can actually... Where did I go? Where did I go? What the hell? What just happened? Okay, there I am. Accidentally pressed Page Down and lost my footing. Okay, print the parent name over here. Print the file name as well, which will be `d.Name`, and that's all we need to do. Now, why is this giving me an error? It's missing a return. Okay, we need to return. We need to return. Where do we need to return? Over here, right? Let's just say `return nil` over here. Okay, the whole thing works. This project folder isn't really necessary anymore. You'll see why, because now that we are creating all of these subfolders, we can comment this whole thing out. Now that we are creating all of these subfolders, this `filepath.WalkDir` will also create the project folder for us. You'll see how that works. Let me make sure that my drive is empty, or actually, we don't even need to delete this previous project because this deal, where is it? This delete previous folder will do that for us. So let's just run this whole thing and see if it works. It's deleting the previous folder. It's creating the subfolder in parent programming. You can see this is the project folder, and it's going to upload all of these files. And oh my God, this video is getting too long as well. Oh my God. Oh my God. Oh my God. What am I going to do about this? See, I create such complex projects that I just have to make long videos, and then I don't like making long videos because they don't get as much engagement as shorter videos anyway. This is the project that we uploaded. You can see over here. Now we have these subfolders. There is no `test.txt` file over here. It's going to be inside of this subfolder. There it is. It has the text over here, so it was uploaded successfully. And we also have this `sub2`, which will have the `subtext` file with its content. It was uploaded successfully, just like I have over here in my path, right? So now we can upload subfolders as well, and we can also delete the previous folder. And with that, the video is getting way too long, so we're going to stop there. And yeah, that's the end of part four, boys. In the next video, I'll probably implement the `.gitignore` stuff over here so that we can ignore all of these files inside of our `.gitignore`, as well as the files in `config.json` that we have in this ignore list, so we don't need to upload all of those files, right? We'll implement that in the next part, next part of the video. That's a pretty simple feature to implement. And yeah, that's the end of part four. Like, comment, subscribe, all of those things, and see you in the next video. Thank you for watching. And yeah, I'm not really good with outros, with video outros. So I'm just going to say bye-bye. Have a nice day. And yeah, that's all.