Syncing projects with Google Drive - Part 16 (Custom gitignore parser)

Good morning, guys. Welcome back to the DriveSinker project. In the previous video, we fixed a couple of bugs. I'm not going to go through them right now. But in the previous video, we fixed a couple of small bugs throughout the project. Stuff like making sure that we check the folder name in the project exists function over here. Stuff like adding this special case in the progress bar rendering, where if total files are equal to zero, then we actually, then we don't crash the program, basically. In this video, we're going to start building the git ignore parser that I've been hinting at in the previous video. And so this is the logic that we're going to follow. So let me zoom in as well, just to make sure you guys can see this. The first step is obviously to read the git ignore file. Yeah, obviously. And this is a two-step process. The first step is to ignore all of the invalid lines, lines that are just empty lines to start with, a hash symbol. This means a comment in the git ignore file. So stuff like, where's my git ignore file? Let me show you guys an example. If I have an empty line over here, and then I have another file over here, right? I want to be able to ignore these empty lines. Also, if I have a comment like this is a sensitive file, this is just a comment, right? And so there's no need to parse this either. So I want to ignore all of these invalid patterns and just make sure we're only parsing the valid patterns that we have. And as soon as we find a valid pattern, we're going to append it to the ignore list over here. This is to be a list of strings, and we're going to process all of these patterns and figure out which files to ignore, which files to allow. And once we're done reading the git ignore file, we're going to start the folder upload. So in our code, this means that we're going to, in here, in the upload folder function, we're going to start the walk der function, and we're going to walk through the entire project folder, upload all the files, right? That's what this, that's what this box is supposed to mean. And then we're going to run a function called should ignore on each path, right? So every single time we get a path in this walk there function over here, we're going to check whether we should ignore it or we or if we should allow it, right? Once that's done, we're going to match each possible pattern. And in order to find out whether we should ignore a file or allow it to be uploaded, we need to match every single possible pattern against that file. So a quick note over here: these rules can be mixed together, so we can't just use a single switch statement to match them, right? Multiple rules can be combined in the same line. I can have stuff over here that combines multiple patterns, stuff like, like I showed you in the previous example, right? In the previous video, sorry. We can have a double star over here to match all folders and subfolders. We can then have a folder name inside over here, like the pycash folder in Python projects, and we can have another pattern over here, like a wild card that says ignore all of the files inside the pycash folder. So this is combining like two or three different patterns of a get ignore file, and it's combining them all in the same line, right? So we want to be able to parse all of these patterns in the same line. So we can't use something like, we can't use something like a switch statement over here. Uh, let me let me show you an example. Right here, I'm going to remove this in just a bit. But we can't say something like switch pattern and then case one, case two. We can't do stuff like that, right? Because multiple patterns can be on the same line. And then here we have all the patterns. We can have a pattern like slash foo, which means we need to match this path from the root directory. We can have just foo, which means this can be at any depth, meaning this folder or this file foo can be in any folder or subfolder in our project, and we'll still ignore it. We can have double star slash foo, and that's that's the same thing as this empty foo over here. It's just going to ignore foo, the folder or the file, in every single folder or subfolder. And finally, we also have foo slash double star, and this means that foo is a directory, not a file, and we going to ignore everything inside that directory. So simple enough. And by the way, there are a couple more patterns over here which I didn't really draw out, but there something like you can, you can have something like a double star in between the path. You can also have it at the end, like we have over here, and at the in the beginning, like we have over here. So yeah, we're going to be implementing all of these patterns in the should ignore function. And yeah, that is the entire process, or like an overview of the process. So now that you understand all of that, let's move on to the code and let's start actually implementing this. So I'm going to create a new file over here, call it ignore.go. There we go. And this will be package api, obviously. And we're going to start by creating a new struct over here. I'm going to call it get ignore. Now, this is going to have a list of ignore patterns. This is a new struct that we need to implement, that we need to write. And before I do that, this is also going to have a folder that will be, I guess, models.folder, a pointer to that. Now, this ignore pattern struct is going to be, it's going to have a pattern, which is going to be a string. It'll have a couple booleans. First of all, negate, that will be A boolean negate basically means whether we should negate this pattern or not. Basically, you can have a special exclamation mark in a gitignore file, and then you can have a pattern after that. So let's say I have a file called, I don't know, test.txt. If I have an exclamation mark at the beginning over here, then this is going to allow this file, right? It's not going to ignore it. It's going to allow it. Usually, when we think about gitignore files, we just think about adding patterns that will ignore files, right? Well, this exclamation mark symbol allows us to allow, specifically allow, a file to be uploaded and not ignore it. That's what we call the negate symbol, and that's why I'm adding a boolean over here to check for that. We're also going to have a dir only boolean. So basically, this is going to be a directory, right? Not a file. And we have from root, which is another boolean, and this just tells us whether we should start scanning from the root or scan, or scan at any depth, basically. And finally, a special little field over here called double star mode. This will be a type of double star mode, and I'll create an enum over here to actually hold all of this. Let's say const. You can say leading double stars. There's really only three options over here, right? You can have double stars at the beginning of the pattern or in the middle of the pattern, or you can have them at the, I guess, the end of the pattern, right? So let's say trailing double star and middle double stars, and they should be starts up here as well. There we go. Awesome. So we've defined most of the data types that we're going to use. Let's create a new git ignore function over here. This will take a folder that will be models.folder. It'll take a global ignore, which will be a list of strings. This will be the files that we ignore globally in every single project. This comes from the config.json file where we have this ignore key, and we just ignore this list of folders or files in every single project that we sync, right? That's the global ignore list. And then finally, a boolean over here called read git ignore. So this will be a boolean because remember, we can configure whether we want to read the gitignore file or not. In the config over here, we can set use git ignore to false. I don't know why anyone in their right mind would choose to do that, but yeah, there is an option to do that, so we need to handle that. Okay, and this is going to return a pointer to git ignored. There we go. The very first thing we need to do is we need to convert this global ignore list of strings into a list of ignore patterns over here so we can actually add them to the git ignore struct. So let's say to ignore equals make a list of ignore patterns with a length of zero, but a capacity of the same length as the length in the git ignore list. Then we can loop over the entire list of global ignore, and we can say to ignore and create a new ignore pattern struct over here. Pass in the pattern, which will just be patterned. There we go, and that all we need to do over here. Then we can say if read git ignore is true, then we should do something about that, right? Let's write this in a bit. First of all, down here we can just return the git ignore, a new git ignore struct over here. You can pass in to ignore over here and the folder over here. Now, before I actually write the block of code inside this if conditional over here, we want to create a new function first, and that function is going to read and parse our git ignore file. So let's go create that down here. I'll call it func read git ignore. It'll take a folder that'll be models.folder, and it'll return a list of ignore patterns called patterns and an error called er. Now we can already see an error over here. It says read git ignored redeclared in this block, and this is because we already have a function called read git ignore down here. We're not going to be using it anymore, and so what we should do is we should just comment the entire thing out. But before I actually do that, I'll just copy this entire thing because we are going to use it as like a starting point. Now let's comment this, go back to the ignore file over here, paste this in, save, so it actually imports all of the stuff over here, all of the packages. Right now, we can remove this line down here, and in the scanner, we can say line equals strings.trim space s.text. There we go. So if for some reason in the gitignore file you have a file, a pattern called client.json, and you just add a random space next to it, we're going to ignore that, right? Obviously. Now, I said that the first step was to ignore the invalid patterns, right? Stuff like over here where we skip the invalid lines, empty lines, lines that are comments. So let's do that first. Let's say if the line is an empty string or if strings.has prefix, the line has a prefix of a hash symbol, then we know that this is an invalid line, and we can skip it. So we'll just say continue on to the next line. Cool. Now we can create an empty ignore pattern over here and just pass in the pattern, which will be line. Now we can say if strings.has prefix, this is where we actually start to scan for all of the complex patterns that I mentioned in the beginning of the video. So let's do that. Let's say strings.has prefix p.pattern and an exclamation mark. If this is the case, we'll say p.negate. equals true because this has been negated, and we'll say p.pattern equals line and skip the first character. So basically, we apply the negation over here by setting negate to true, but we also remove the exclamation mark from the actual pattern because we don't need it anymore, right? Next, if strings.HasSuffix, remember guys, this is a suffix, not a prefix, pass in p.pattern, and the suffix, the ending character, should be a slash. If that is the case, then we know that this is a directory and not a file, right? So we'll say p.zero only equals true, and p.pattern will be strings.TrimSuffix, passing the pattern over here, and trim that slash at the end. Now, if we have a prefix that is a slash, then we know that this is supposed to be matched from the root directory, right? So we can say p.fromRoot equals true in this case, and we can say p.pattern equals strings.TrimPrefix(p.pattern, slash), and we just trim that slash at the beginning, right? Again, this is all the stuff that we figured out over here in the flow chart, and I'm just implementing all of the rules and all the different patterns over here in the code right now. Let's say if strings.HasPrefix(p.pattern) and then a double star with a slash, then we'll say p.doubleStarMode equals leading double stars, and let's also remove those stars from the pattern. Now that we actually set the double star mode, we say strings.TrimPrefix from p.pattern, and that be the double star slash. There we go. Then if strings.HasSuffix(p.pattern) and then a slash and double stars, then this is good. This is going to have the double star mode of trailing double stars. There we go, and the pattern will be strings.TrimSuffix, this time not prefix, suffix, p.pattern, slash, double star. Finally, we have strings.Contains(p.pattern) and then double stars anywhere in the pattern, right? If this is the case, then that means we're using middle double stars, right? And now we can clean up the pattern over here as well by saying strings.Replace(p.pattern), replace the double stars with an empty string, and the end can just be one. So that's all we need to do to read the Git ignore file. And down here in the ignore, we can just return patterns, right? And this patterns is just the list that we create over here, right? And of course, we need to actually append to that patterns list by saying patterns.append and passing p over here. Wonderful. Okay, so that's all I'm gonna do for this video. Let's just do a quick recap. We commented out the read git ignore function over here, right, in the sync file. That's the old function. We're not going to use it anymore, okay? And over here, this is obviously going to cause an error in the upload function because this expects the read git ignore file to return a list of strings, right? And that is why we see an error over here. We're going to fix that in the next video because we're still actually building this new git ignore parser, right? But yeah, we created this ignore.go file. We created the GitIgnore struct, the new GitIgnore function, the ignore patterns, the double star mode, and then this read git ignore function, which is basically just an improvement over the old read git ignore function that we had. And this one just parses all of the patterns, all the complex patterns that we mentioned in the beginning of the video, and it just converts them into a format that we can understand, right? It converts them into this ignore pattern struct over here with all of these fields and the booleans filled up so we can actually understand what this pattern is and does in code, right? That's what this read git ignore function is going to do now, and it's going to return those patterns. And then we're going to use them to actually, you know, actually figure out whether we should upload a file or not. Now, lastly, before we end this video, I totally forgot to do this, by the way, in the new git ignore function. In the if read git ignore is equal to true block, let's say git ignore list and error equals read git ignore. Pass in the folder as well. Let's say if error does not equal to nil, we'll say log.printf cannot read git ignore, will upload all files. Okay. And of course, let's print the error as well by saying percentage read and add a new line, and then pass in the error. Now, assuming this worked, we can say to ignore.append git ignore list as such. And that's pretty much all we need to do. Now each time we call this new git ignore function, we're going to read the gitignore files, we're going to parse all the patterns in there, we're also going to append the global ignore list to this ignore main list, right? And we're going to return that gitignore struct, and we can then use that to figure out which files to upload, which files to not upload. We're going to continue building this in the next video. Stay tuned for that. This video is done. Thank you for watching. Thank you for following along with the project, and stay tuned for the next one. Like, comment, subscribe, share, do all of those wonderful things. And lastly, quick shout out to myself. I'm a freelance developer, and I build websites, apps, software, and stuff like that. So if this is something you're interested in, then check out my website. Link will be in the description. You can also book a call with me for free, and you can hop on a call, and you can discuss your project. Again, for free. And if you want to work with me on some project or something, this is the best way to get started. That's all for today, guys. Thank you so much for watching, and I will see you in the next video. Next video