Syncing projects with Google Drive - Part 11 (Mapping local to remote projects)

All right, Drive Syncer. Quick recap: In the previous video, we basically just fixed an issue with our progress bar that was exiting prematurely before reaching 100. And more importantly, we started building this tracker struct over here to track all of the file changes in our local folder on Google Drive and so on and so forth, so we can actually only upload the files that were actually modified instead of uploading every single file, regardless of whether it was modified or not. And this is obviously not finished yet, so we're gonna work on that now. You'll notice that there's still plenty of problems with this tracker that we've built so far. First of all, the most important issue is that it's uploading duplicate files, and the reason it's doing this is because we're not deleting the previously uploaded files over here. And so we need to change that. We need to start tracking the files so that we can actually ignore the files that have already been uploaded and only upload the files that are modified, right? And of course, make sure to delete their previous files so we don't end up with having duplicates and stuff. So that's what we're going to work on now. Let me also zoom in so you guys can actually see, and yeah, let's get started. So the very first thing we want to do is build a function called `walkProjectDirectory`. So this will be another pointer receiver, I guess. Let's say `walkProjectTree`. And this function is going to walk through the entire project on Google Drive and basically kind of like make a whole tree-like structure of the entire project on Google Drive, right? And it is going to be kind of a recursive function, so might be a bit confusing to figure out, but let's do this. First of all, let's just create a list of nodes, a map of nodes. This is going to be a map with string keys and node pointers as the values. And these nodes are just the node struct that we defined in the previous video. We're actually going to use these in this video. And then we need a list of folders, right, to actually walk through. So this is going to be like a queue. I'm just going to call it `queue`, but it's going to be a list of folder IDs. And the first folder that we need to walk through is obviously going to be our project folder. So I'll put the folder ID over here, and we're going to start with this. We're also going to need to track the folder path of every single folder that we go through in this queue. So to do that, let's create another list of strings, or actually, instead of a string list, let's use a map over here that can hold strings. There we go. And then the first folder in this map is just going to be our project folder, and the value will just be the path. Oh, sorry, the key will be the folder ID, and the value will be the folder path. You'll see why we need this in just a second. Now let's say `for length of q is greater than zero`. Basically, as long as there's a folder inside the queue, we're going to continue running this. Right now, let's say the current folder that we want to access is going to be the first element of the queue, and then the new queue is going to be `q` and every element except that first element. So since we're processing it, we're also going to remove it from the queue, right? And then I also want to get the base path of the current folder that we're in, right? And this is why I actually need this folder paths map, so we can actually get the base path of this folder inside our project, inside like our local project directory, right? Now it's just going to have the current project folder, right? But once we start to go through subfolders and stuff, we're going to need their base paths, and that is why we need to have this folder paths map as well over here. Also, let's also handle, what do you call it, pagination. So let's create a page token. This will be empty for now, and Google Drive API will automatically fill this up for us. We don't need to do anything over here. But just in case there's like hundreds of files in your project folder, you want to actually make sure that you're handling pagination here as well. And just for now, I want to have a bunch of print statements over here as well, so I can actually tell what's going on. I'll say, "Walking the base path," just so we know what's going on, right? And now let's create an infinite loop over here to actually get all of the files in this folder. First of all, we're going to need to write a query, a search query. So let's say `fmt.Sprintf`. Now, we want to make sure that the files we're searching for over here have the current folder that we're processing in the queue right now, have that current folder as their parent folder, right? And obviously, make sure that they're not trashed, right? Otherwise, why are we even searching for them, right? Now let's say request. Where is my Drive API? There it is. `.Files.List().Fields`, and then we'll say files. And we're going to need a couple of fields over here. First of all, obviously, the ID, the name, the MIME type, but also the size, modified time, the SHA-256 checksum, and the list of parents. We're going to need all of these. And then let's also pass in our search query as well. There we go. And then let's also handle the pagination over here. If the page token is not an empty string, then we should add the page token to this request, and then we're ready to actually send this request to Google Drive servers, right? Let's handle the error as well. I'm not going to do anything fancy over here for now. Let's just print the error and return if there is one, right? Instead of returning, we can also just break out of here and continue walking through the other project directories. But yeah, with that done, now let's access the list of files over here. I'll say `for f in range r.Files`. And this will give us the list of files from the Google Drive API. Now, if this is a folder, and we can tell that from the MIME type, if the MIME type matches our folder MIME type, then that means this is a folder on Google Drive, right? And if that's the case, then we need to obviously add this to the queue and make sure we walk this folder as well, right? So we'll add it to the folder paths by saying `f.ID` over here, and then we'll say `filepath.Join` the base path along with this folder's name, let's say. And this will create the local path of this folder, right? Local meaning the path on your machine, right? That's how you actually create like a tree of the entire project directory. Now, to process this folder, we also need to add it to our queue, right? So let's say `queue.Append` and add the folder ID over here, and this will make sure that we actually process this in the queue once we get to it, right? Then once we're done with that stuff, let's access the folder of this file or folder, whatever it is, by creating a folder, by creating a parent variable. And this will create the, this will hold the folder ID of the parent. And then let's say if the length of `f.Parents` is equal to one, then we can say the parent will be equal to `f.Parents` and the first element in there. And we do it this way because there's no guarantee if the file or folder has a parent over here, right. And the reason I do it this way is to just make sure that there is actually a parent in that list, right? Otherwise, it could cause some sort of a runtime crash in our program. And we obviously don't want that, right? Now, I also want to get the local path of this file. Basically, we're getting all of these files from Google Drive, right? But they also exist on our local machine, right? The project files and everything exist on our local machine as well. So I want to create the local path of this file from Google Drive as well. To do that, just join the base path with this file's name. Now, before we move any further, we should finish creating this node struct over here because right now, as you can see, it's completely empty. So let's finish building this as well because we're going to need a ton of fields over here. So first of all, the ID. This will be the file ID on Google Drive. And then the name as well, the MIME type, which will be a string as well. And then is folder. This will be a boolean, and we need to differentiate between files and folders on Google Drive. And obviously, we can use the MIME type to do that, but this is going to be a boolean over here. Also, the file size. This will be in bytes. The modified time. This will be a string. The checksum, the SHA-256 checksum. And this is what we will use to verify the file integrity. And finally, the parent as well. Alright, so that's all we need over here. Let's go back up over here in the walk function. Now let's say nodes and pass in the local absolute path. And then create a node struct over here, which will be a reference, of course. And then the ID will just be f.id. The name will be f.name. The MIME type will be f.mime type. Is folder, this will just be a conditional over here. We'll say the MIME type equals folder MIME type. If that's true, it's a folder, and it will be assigned true over here in this boolean field. Modified time, f.modified time. Not modified by me time, modified time. Checksum, f.sha256 checksum. Finally, the parent, which will just be parent over here. Now, outside of this loop over here, where we're looping over the files, let's check if there is another page that we need to crawl. And if there isn't, if Google Drive does not return a page token for the next page, that means there are no pages after this, and we can safely break out of this loop. Otherwise, if there is another page to crawl, then we can say page token equals r.next page token, and then we can just continue running this loop until we run out of pages to crawl, right? And that's pretty much all we need to do to walk through the entire project tree. Right now, outside over here, outside all the loops, but still inside the function, let's say t.nodes equals nodes. This, the nodes field on the tracker object. And then, just for debugging, just to make sure that we wrote all of that code correctly and there are no mistakes, let's loop through the entire nodes list. We'll say range t.nodes, and then let's println the key and then separate both of these with an equal to sign, and then v.id. And then let's just print something like, I don't know, a couple of separators over here, and then let's say project 3. And because really, I just want to make sure that this is all working. Right now, since we built this, we also need to integrate this. To do that, we go over to the sync file over here, API sync, and where is it? Down here, where we check whether the project exists, right? If the project already exists, then we need to walk the project directory. So here, I'll add a log statement over here as well. I'll say the folder name, and then we'll say project already exists, walking project tree. And of course, add a new line as well. And then let's say tracker.walk project tree. And then let's do a quick review of this function. Just make sure we didn't break anything. We didn't add anything stupid. Let's add this statement up top over here as well, so we know when it actually starts to, you know, walk the project tree. And then let's run this program and see if it actually works, see what happens. Okay, so it looks like it walked through the entire project tree, and now it's uploading all of the folders and files. So while it does that, let me just look at the project tree over here. Okay, so these are two separate projects that we uploaded. So let's go all the way to the top over here. Yeah, this is a bit of a confusing output to look at because it's uploading two different projects at the same time, and since this is happening concurrently, it's just, you know, messing up all of my output, right? So let me go over to config and just sync only one project at a time. So let's comment this one out. Let's remove this comma. Okay, so apparently, you also can't have comments in JSON, so let's just remove it completely. Clear everything over here, and then sync again. Okay, so here we start. It's walking through the entire project tree. It's walking through all of these folders, and here are the list of nodes along with their file IDs on Google Drive. And of course, this also has the folders as well, like sub, like API, and so on and so forth, right? So now we have every single file and folder in this project on Google Drive, along with their file and folder IDs. So we can actually access them directly on Google Drive, and also along with their local path. I can literally use this local path to open up this same file on my local machine right here. So now we have a clear mapping between our local files and folders and the files and folders on Google Drive. And we can now use this to track which files have been uploaded, which have not, which have been modified, which need to be re-uploaded, which need to be deleted, and so on and so forth. And of course, after scanning the project tree, it's going to upload everything, create the subfolders and all of that stuff. Right now, this video is getting a bit too long, like most of my videos, so I'm going to stop over there. And in the next video, we're going to continue building this tracker functionality. We've built the functionality to walk the project tree. So the next thing we need to do is to just add, like, some sort of a should upload function, a function that tells us whether we should upload a file or not, and then integrate that in our sync algorithm. And then we will use that to decide whether to upload a file or not, right? And after that, we should have the sync algorithm perfectly implemented. So that's coming in the next video. In this video, we just wanted to build the walk project tree function and walk through the entire project tree and create a mapping between the local project and the project on Google Drive. Hope you learned something in this video. This project is coming along very nicely. If you did, leave a like, subscribe, comment, share, all of those wonderful things, and I'm gonna see you in the next video. Bye!