Syncing projects with Google Drive - Part 5 (Reading .gitignore files)

Let me just quickly recap what we did last time. So I'm not even sure what we did last time, but we basically built all of the upload functionality over here to upload files and folders to Google Drive. We also built a couple functions over here to delete the previous folder so we don't have duplicates in our Drive, and also to create subfolders so we have a hierarchy of folders and subfolders in our projects, right? So this is very close to completion already. This is already something you can put in production and actually use in the real world, right? But there's a couple more features I want to add. So first of all, I want to add the ability to read Git ignore files. Basically, I want to be able to, you see this Git ignore file over here, I want to be able to read the Git ignore file of every single project and then based on that decide which files and folders we're going to upload and which ones we're going to ignore, all right? Because we don't want to upload stuff like this, the token, the client, the authorization tokens, and sensitive secrets like that onto Google Drive, right? We want to keep these things local to our machine. Basically, anything that you're not going to upload to GitHub or source control, you shouldn't upload to Drive or any other backup solution either, right? So we're going to do that today in this video. Let's go back to sync.go. This is where I'm going to build the entire thing. Let me just walk you through what we're going to do. There's two parts to this. So first of all, let me open up config.json, and over here you can see the ignore keyword, right? We want to basically be able to use this ignore list over here. This is the global ignore. This is going to take effect in every single project. Basically, in every single project that we synchronize, we want to ignore the node modules, the vent, this is the Python environment, virtual environment, the dot Git folder. We don't need to upload this either, right? This stuff will be ignored in every single project that we upload, right? Along with that, so this is one ignore list, right? Along with that, we want to also be able to use the Git ignore files in every single project. And this is the second ignore list. So there's going to be two lists of files and folders that we want to ignore in every single upload, right? And we're basically just going to build those two lists and then combine them and then just make sure we're not uploading them, that we're skipping them, right? So let's get started. The very first thing I'm going to do is all the way down here at the end of the file, create a function called readGitIgnore. This will take a folder, which will just be appointed to the folder object, and it will return two things. First, the list of strings of file names that we need to ignore, and second, an error over here, all right? Now let's do f and error equals os.Open. Let's open up the Git ignore file. The file path for this will be filepath.Join. We'll say join the folder path with .gitignore. So we're saying every single Git ignore file is going to be inside the folder at the root of the folder, right? This is how usually Git ignore, this is how Git source control works, right? So yeah, we'll create the path and open up the file. We'll make sure to handle the error as well. Let's just say return nil and error. And of course, make sure to use defer to close the file when you're done with it. Now let me just quickly show you the Git ignore file again. What this basically looks like is we have a bunch of file names and folder names per line, right? So on every single line, you have one folder or one file name, right? So we want to be able to scan this per line, right? Let's do that. To do that in Go, we can use something called bufio.Scanner. I'll say new scanner over here. Pass in the reader. This will just be the file. And we'll call this s. And then we'll say for s.Scan. And then we'll say append to ignore. Append. And we will append s.Text. So this will scan every single line in the Git ignore file and return the text of it, right? Return it as a string. Now, we also want to make sure that we use, that we clean up this string by using strings.TrimSpace. And this will return the string with all leading and trailing whitespace removed, right? So basically, I don't want to have something like, so we have client.json in Git ignore, right? I don't want to be able to add a bunch of spaces over here and empty characters and scan those as well, right? Because that would just break the entire thing, right? So we make sure to remove all of this trailing and leading whitespace, clean up the string, and then that's when we append that to the ignore list over here, right? And if there's an error, let's say if error equals s.Err and error does not equal to nil, then we return nil and we return the error as well. Otherwise, we return the to ignore list and we return nil for the error. And that's all we need to do. Now with this function, we can read the Git ignored files in every single project. Now back up in the upload folder function, now we need to create the actual to ignore lists over here, right? So first of all, first of all, let's create the global ignore list, right? The one over here in this ignore keyword in config.json, right? To build this, let's just say to ignore equals make, and we will say make a list of strings. I ruined the syntax. Make a list of strings with a length of zero, but a capacity for what's the capacity for drive.config.ignore over here. We'll say length of drive.config.ignore. This is kind of like an optimization thing. Basically, I'm saying make a list of strings. That's what we need, with a length of zero because we don't want a length right now, but a capacity for the amount of elements that we have in the ignore list in the config, right? This ignore list over here, right here. If I instead set the length of this list to that, then it would basically fill it up with empty elements, right? Which we don't need to do, right? But instead, if I set the capacity to that list, right, then it will allow us to have that much capacity so we won't need to reallocate space in this list to add new elements, but we also won't need to fill it up with empty elements because we're setting length to zero. So yeah, a bit of a minor optimization that I'm doing over here. Anyway, we'll say append to ignore, and then we'll say drive.config.ignore and use this spread operator over here to add everything in the ignore list in the config to this ignore list that we're creating over here. So with these two lines of code, we basically have the global ignore list over here all set. Now we need to add the Git ignore list, right? So to do that, let's do a quick check. Let's say `drive.config.useGitIgnore`. If this is true, then we should add the Git ignore list. Otherwise, not, right? So, a quick check to make sure that that's true, and then we can just say something like `gitIgnore` and `error` equals read Git ignore and pass in the folder over here. Make sure to handle the error as well as usual. We'll say `log.Printf("Cannot read Git ignore: %v\n", error)`, and we'll upload all files and then print the error as well over here. Make sure to add a new line as well. Let's say error over here, and that's all we need to do. Awesome. So, if we cannot read the Git ignore, then we'll just upload all files by default, right? And then we can just say `toIgnore = append(toIgnore, gitIgnore)`. There we go. And now we have everything we need, basically. Now all we need to do is loop through this `toIgnore` list and just discard everything that matches over here, right? And if we do that, then we're basically done with this feature. So, to do this, let's head into the `filepath.WalkDir` function, and up top over here, I'm just going to loop over the list. I'll say `for _, ignore := range toIgnore`, and let's grab the ignored directory as well, the full path of this directory, which will be `filepath.Join(folder.path, ignore)`. There we go. So basically, if our ignored folder is something like `client.json`, then when we add the folder path to it, we'll have an absolute path to it, right? Like something like `C:/project/folder` that we're trying to sync, slash `client.json`, right? We want to have an absolute path over that, right? So that's what I'm doing over here by using `filepath.Join`, and then let's create a conditional over here. Let's say `shouldSkip = ignoredDir` and check if the path equals to that. So we're saying if the path that we're currently on in this `WalkDir` function equals the `ignoredDir`, right? If the current path that we're on equals the directory that we're trying to ignore, then you should skip that directory. Now, along with this, let's add an OR conditional as well. We want to add a bit of a, we want to handle a little edge case as well over here. Let's say `strings.HasPrefix(path, ignoredDir + string(os.PathSeparator))`. And let's convert this path separator to a string as well over here. And there we go. So basically, let me walk you through the entire conditional real quick. First of all, I'm saying if the path equals the `ignoredDir`, right? So if the current path that we're on equals the directory that we're trying to ignore, then we should obviously skip it, right? That makes sense. Now, along with that, we're checking if the current path that we're on has the prefix of the ignored directory plus a path separator. So basically, we want to be able to ignore the directory even if we're on a subfolder of that directory, right? Something like, if I have something like a, if I want to ignore, let me do this in comments as well. There we go. If I want to ignore a folder called, I don't know, `ignored`, right? That's the folder name. And if I'm on the path something like `project/ignored/file.txt`, right? If I on this path, then this conditional right here is not going to match, right? Because the path is going to be all the way to `file.txt`, and the ignore directory is going to be `project/ignored`, right? So this is not going to match, right? Which is why we need to handle the edge case over here by using `strings.HasPrefix(path, ignoredDir)`, right? If the path we're currently on, which is this, has the prefix of `project/ignored` plus a path separator like this slash over here, then we can safely ignore that as well, right? So that's why I add this little extra condition over here, right? Let's remove these comments, and now we just use this condition over here, right? We just say `if shouldSkip`, we'll say `log.Printf("Skipping %s\n", path)`, and I'll print the directory that we're skipping over here. Let's say `ignoredDir`, or actually, we can just say `path` over here because it's just going to be the same thing, right? And of course, make sure to add a new line as well over here. And now one more thing. We can say if this is a directory, then we should return, what was it? There was a special error, `filepath.SkipDir`, and if it is not a directory, if it's just a regular file, then you can just return `nil` over here, right? So basically, `filepath.SkipDir` is a special error code over here, and if we return this, then it will basically skip walking through the entire directory, right? It won't try to access all the other files in this directory if we're going to just ignore the entire thing, right? So this is very useful as well. Otherwise, if it's just a file, then we just return `nil`, and it can move on to the next file, right? And that's all we need to do. That is all we need to do. So, let's test this out now. So here I am in my Google Drive, and you can see I have the `client.json` and the `token.json` uploaded over here, right? Which we don't want. So what I'm going to do is I'm going to run the entire thing again, and this time it should read the Git ignore file and refrain from uploading all of these sensitive files, right? So let's try running this. Okay, the token is expired. That makes sense. I want to be able to, this is a separate feature, by the way, but I want to be able to refresh the tokens directly, automatically, without me having to, like, do it myself, right? But I think I'll build that in a different video. For now, let's just say `forceRefresh = true` and just run this thing over here. There we go. Once you add access to this app, it's gonna redirect to localhost, which doesn't mean anything, and you can just grab this whole code over here in the URL and just take that and paste it over here. And there we go. Now it's starting the process. Deleted the previous folder, created subfolders, skipping the Git folder, skipping the `client.json` file over here. As you can see, it's all working, right? And I think it's almost done. Yeah, it's done, and it also skipped uploading the `token.json` file. So perfect, this works perfectly. Let's see what happened on Google Drive as well. There we are. Let's open up this folder, and you will see, yeah, see, there is no `client.json`, no `token.json`, none of those files over here, right? So we were able to ignore all of the files that we don't want to upload and just upload the ones we do want to upload, right? So, yeah, this whole thing works, and with that, as you can see, this whole feature works, and we're done with the video. You can go around and play with this a bit more, maybe add a couple more features, I don't know, but yeah, this whole thing works. In the next video, I'm probably going to