> ## Content Index > Fetch the complete content index at: https://linuxhandbook.com/llms.txt > Use this file to discover other available public pages before exploring further. # csplit: A Better Way to Split File in Linux Based on its Content - URL: https://linuxhandbook.com/csplit-command/ - Published: 2018-07-16T09:06:25.000Z - Updated: 2022-11-13T01:45:47.000Z - Description: How to split a file in Linux based on its content? Learn some practical examples of the GNU coreutils csplit command. It's more useful than the popular split command. - Author: Sylvain Leroux - Tags: Commands When it comes to splitting a text file into multiple files in Linux, most people use the split command. Nothing wrong with the [split command](https://linuxhandbook.com/split-command/) except that it relies on the byte size or line size for splitting the files. This is not convenient in situations where you need to split files based on its content, instead of size. Let me give you an example. I manage my scheduled tweets using [YAML](https://gist.githubusercontent.com/s-leroux/42377061175b9c323bf5feb2d81bdb65/raw/tweets.yaml?ref=linuxhandbook.com) files. A typical tweet file contains several tweets, separated by four dashes: ``` ---- event: repeat: { days: 180 } status: | I think I use the `sed` command daily. And you? https://www.yesik.it/EP07 #Shell #Linux #Sed #YesIKnowIT ---- status: | Print the first column of a space-separated data file: awk '{print $1}' data.txt # Print out just the first column For some unknown reason, I find that easier to remember than: cut -f1 data.txt #Linux #AWK #Cut ---- status: | For the #shell #beginners : [...] ``` When importing them into my system, I need to write each tweet to its own file. I do that to avoid registering duplicate tweets. But how to split a file into several parts based on its content? Well, probably you can obtain something convincing [using awk commands](https://linuxhandbook.com/awk-command-tutorial/): ``` sh$ awk < tweets.yaml ' > /----/ { OUTPUT="tweet." (N++) ".yaml" } > { print > OUTPUT } > ' ``` However, despite a relative simplicity, such a solution is not very robust: for example, I didn’t properly close the various output files, so this might very well reach the open files limit. Or what if I forgot the separator before the very first tweet of the file? Of course, all that can be handled and fixed in the AWK script, at the expense of making it more complex. But why bothering with that when we have the `[csplit](https://linux.die.net/man/1/csplit?ref=linuxhandbook.com)` tool to accomplish that task? ## Using csplit to split files in Linux The `csplit` tool is a cousin of the `[split](https://linux.die.net/man/1/split?ref=linuxhandbook.com)` tool that can be used to split a file into **fixed-size* chunks. But `csplit` will identify the chunk boundaries based on the file content, rather than using byte count. In this tutorial, I’ll demonstrate csplit command usage and will also explain the output of this command. So, for example, if I want to split my tweet file based on the `----` delimiter, I could write: ``` sh$ csplit tweets.yaml /----/ 0 10846 ``` You may have guessed the `csplit` tool used the regex provided on the command line to identify the separator. And what could be those `0` and `10983` result displayed on the standard output? Well, they are the size in **bytes* of each created chunk of data. ``` sh$ ls -l xx0* -rw-r--r-- 1 sylvain sylvain 0 Jun 6 11:30 xx00 -rw-r--r-- 1 sylvain sylvain 10846 Jun 6 11:30 xx01 ``` Wait a minute! Where those `xx00` and `xx01` filenames are coming from? And why `csplit` split the file into **two chunks only*? And why the first data chunk has a length of **zero bytes*? The answer to the first question is simple: `xxNN` (or more formally `xx%02d`) is the default filename format used by `csplit`. But you can change that using the `--suffix-format` and `--prefix` options. For example, I could change the format to something more meaningful for my needs: ``` sh$ csplit tweets.yaml \ > --prefix='tweet.' --suffix-format='%03d.yaml' \ > /----/ 0 10846 sh$ ls -l tweet.* -rw-r--r-- 1 sylvain sylvain 0 Jun 6 11:30 tweet.000.yaml -rw-r--r-- 1 sylvain sylvain 10846 Jun 6 11:30 tweet.001.yaml ``` The prefix is a plain string, but the suffix is a format string like the one used by the standard C library `[printf](https://linux.die.net/man/3/printf?ref=linuxhandbook.com)` function. Most characters of the format will be used verbatim, except for conversion specifications which are introduced by the percent sign (`%`) and which ends with a conversion specifier (here, `d`). In between, the format may also contain various flags and options. In my example, the `%03d` conversion specification means: - display the chunk number as a decimal integer (`d`), - in a three characters width field (`3`), - eventually padded on the left with zeros (`0`). But that does not address the other interrogations I had above: why do we have **only two* chunks, one of them containing **zero* bytes? Maybe do you already have found the answer to that latter question by yourself: my data file starts with `----` on its very first line. So, `csplit` considered it as a delimiter, and since there was no data before that line, it created an empty first chunk. We can disable the creation of zero bytes length files using the `--elide-empty-files` option: ``` sh$ rm tweet.* rm: cannot remove 'tweet.*': No such file or directory sh$ csplit tweets.yaml \ > --prefix='tweet.' --suffix-format='%03d.yaml' \ > --elide-empty-files \ > /----/ 10846 sh$ ls -l tweet.* -rw-r--r-- 1 sylvain sylvain 10846 Jun 6 11:30 tweet.000.yaml ``` Ok: no more [empty files](https://linuxhandbook.com/empty-file-linux/). But in a sense, the result is worst now, since `csplit` split the file in just **one* chunk. We barely can call that “splitting” a file, can’t we? The explanation for that surprising result is `csplit` does **not* at all assume each chuck should be split based on the **same* separator. Actually, `csplit` requires you to provide each separator used. Even if it is several times the same: ``` sh$ csplit tweets.yaml \ > --prefix='tweet.' --suffix-format='%03d.yaml' \ > --elide-empty-files \ > /----/ /----/ /----/ 170 250 10426 ``` I’ve put three (identical) separators on the command line. So, `csplit` identified the end of the first chunk based on the first separator. It leads to a zero bytes length chunk that was elided. The second chunk was delimited by the next line matching `/----/`. Leading to a 170 bytes chunk. Finally, a third 250 bytes length chunk was identified based on the third separator. The remaining data, 10426 bytes, were put into the last chunk. ``` sh$ ls -l tweet.???.yaml -rw-r--r-- 1 sylvain sylvain 170 Jun 6 11:30 tweet.000.yaml -rw-r--r-- 1 sylvain sylvain 250 Jun 6 11:30 tweet.001.yaml -rw-r--r-- 1 sylvain sylvain 10426 Jun 6 11:30 tweet.002.yaml ``` Obviously, it wouldn’t be practical if we had to provide as many separators on the command line as there are chunks in the data file. Especially since that exact number is usually not known in advance. Fortunately, `csplit` has a special pattern meaning **“repeat the previous pattern as much as possible.”* Despite its syntax reminding the star quantifier in a regular expression, this is closer to the [Kleene plus](https://en.wikipedia.org/wiki/Kleene%5Fstar?ref=linuxhandbook.com#Kleene%5Fplus) concept since it is used to repeat a separator that has **already* been matched once: ``` sh$ csplit tweets.yaml \ > --prefix='tweet.' --suffix-format='%03d.yaml' \ > --elide-empty-files \ > /----/ '{*}' 170 250 190 208 140 [...] 247 285 194 214 185 131 316 221 ``` And this time, finally, I have split my tweet collection into individual parts. However, does `csplip` have some other nice “special” patterns like that? Well, I don’t know if we can call them “special”, but definitely, `csplit` understand more of patterns. [How to Split String into Array in Bash \[Easiest Way\]In this quick tip, you’ll learn to split a string into an array in Bash script.![](https://linuxhandbook.com/content/images/size/w256h256/2021/08/Linux-Handbook-New-Logo.png)Linux HandbookAbhishek Prakash![](https://linuxhandbook.com/content/images/2020/06/bash-shell-logo.jpg)](https://linuxhandbook.com/bash-split-string/) ### More csplit patterns We’ve just seen in the preceding section how to use the ‘{\*}’ quantifier for unbound repetitions. However, by replacing the star with a number, you can request an exact number of repetitions: ``` sh$ csplit tweets.yaml \ > --prefix='tweet.' --suffix-format='%03d.yaml' \ > --elide-empty-files \ > /----/ '{6}' 170 250 190 208 140 216 9672 ``` That leads to an interesting corner case. What would append if the number of repetition exceeded the number of actual delimiters in the data file? Well, let’s see that on an example: ``` sh$ csplit tweets.yaml \ > --prefix='tweet.' --suffix-format='%03d.yaml' \ > --elide-empty-files \ > /----/ '{999}' csplit: ‘/----/’: match not found on repetition 62 170 250 190 208 [...] 91 247 285 194 214 185 131 316 221 sh$ ls tweet.* ls: cannot access 'tweet.*': No such file or directory ``` Interestingly, not only `csplit` reported an error, but it also removed **all* the chunk files created during the process. Pay special attention to my wording: it **removed** them. That means the files were created, then, when `csplit` encountered the error, it deleted them. In other words, if you already have a file whose name looks like a chunk file, it will be removed: ``` sh$ touch tweet.002.yaml sh$ csplit tweets.yaml \ > --prefix='tweet.' --suffix-format='%03d.yaml' \ > --elide-empty-files \ > /----/ '{999}' csplit: ‘/----/’: match not found on repetition 62 170 250 190 [...] 87 91 247 285 194 214 185 131 316 221 sh$ ls tweet.* ls: cannot access 'tweet.*': No such file or directory ``` In the above example, the `tweet.002.yaml` file we’ve manually created was overwritten, then removed by `csplit`. You can change that behavior using the `--keep-files` option. As its name implies it, it will not remove chunks csplit created after encountering an error: ``` sh$ csplit tweets.yaml \ > --prefix='tweet.' --suffix-format='%03d.yaml' \ > --elide-empty-files \ > --keep-files \ > /----/ '{999}' csplit: ‘/----/’: match not found on repetition 62 170 250 190 [...] 316 221 sh$ ls tweet.* tweet.000.yaml tweet.001.yaml tweet.002.yaml tweet.003.yaml [...] tweet.058.yaml tweet.059.yaml tweet.060.yaml tweet.061.yaml ``` Notice in that case, and despite the error, `csplit` didn’t discard any data: ``` sh$ diff -s tweets.yaml <(cat tweet.*) Files tweets.yaml and /dev/fd/63 are identical ``` But what if there are some data in the file I want to discard? Well, `csplit` has some limited support for that using a `%regex%` pattern. ### Skipping data in csplit When using a percent sign (`%`) as the regex delimiter instead of a slash (`/`), `csplit` will **skip* data up to (but not including) the first line matching the regular expression. This may be useful to ignore some records, especially at the start or the end of the input file: ``` sh$ # Keep only the first two tweets sh$ csplit tweets.yaml \ > --prefix='tweet.' --suffix-format='%03d.yaml' \ > --elide-empty-files \ > --keep-files \ > /----/ '{2}' %----% '{*}' 170 250 sh$ head tweet.00[012].yaml ==> tweet.000.yaml <== ---- event: repeat: { days: 180 } status: | I think I use the `sed` command daily. And you? https://www.yesik.it/EP07 #Shell #Linux #Sed #YesIKnowIT ==> tweet.001.yaml <== ---- status: | Print the first column of a space-separated data file: awk '{print $1}' data.txt # Print out just the first column For some unknown reason, I find that easier to remember than: cut -f1 data.txt #Linux #AWK #Cut ``` ``` sh$ # Skip the first two tweets sh$ csplit tweets.yaml \ > --prefix='tweet.' --suffix-format='%03d.yaml' \ > --elide-empty-files \ > --keep-files \ > %----% '{2}' /----/ '{2}' 190 208 140 9888 sh$ head tweet.00[012].yaml ==> tweet.000.yaml <== ---- status: | For the #shell #beginners : « #GlobPatterns : how to move hundreds of files in not time [1/3] » https://youtu.be/TvW8DiEmTcQ #Unix #Linux #YesIKnowIT ==> tweet.001.yaml <== ---- status: | Want to know the oldest file in your disk? find / -type f -printf '%TFT%.8TT %p\n' | sort | less (should work on any Single UNIX Specification compliant system) #UNIX #Linux ==> tweet.002.yaml <== ---- status: | When using the find command, use `-iname` instead of `-name` for case-insensitive search #Unix #Linux #Shell #Find ``` ``` sh$ # Keep only the third and fourth tweets sh$ csplit tweets.yaml \ > --prefix='tweet.' --suffix-format='%03d.yaml' \ > --elide-empty-files \ > --keep-files \ > %----% '{2}' /----/ '{2}' %----% '{*}' 190 208 140 sh$ head tweet.00[012].yaml ==> tweet.000.yaml <== ---- status: | For the #shell #beginners : « #GlobPatterns : how to move hundreds of files in not time [1/3] » https://youtu.be/TvW8DiEmTcQ #Unix #Linux #YesIKnowIT ==> tweet.001.yaml <== ---- status: | Want to know the oldest file in your disk? find / -type f -printf '%TFT%.8TT %p\n' | sort | less (should work on any Single UNIX Specification compliant system) #UNIX #Linux ==> tweet.002.yaml <== ---- status: | When using the find command, use `-iname` instead of `-name` for case-insensitive search #Unix #Linux #Shell #Find ``` ### Using offsets while splitting files with csplit When using regular expressions (either `/…​/` or `%…​%`) you can specify a positive (`+N`) or negative (`-N`) offset at the end of the pattern so `csplit` will split the file **N* lines after or before the matching line. Remember, in all cases, the pattern specifies the **end* of the chunk: ``` sh$ csplit tweets.yaml \ > --prefix='tweet.' --suffix-format='%03d.yaml' \ > --elide-empty-files \ > --keep-files \ > %----%+1 '{2}' /----/+1 '{2}' %----% '{*}' 190 208 140 sh$ head tweet.00[012].yaml ==> tweet.000.yaml <== status: | For the #shell #beginners : « #GlobPatterns : how to move hundreds of files in not time [1/3] » https://youtu.be/TvW8DiEmTcQ #Unix #Linux #YesIKnowIT ---- ==> tweet.001.yaml <== status: | Want to know the oldest file in your disk? find / -type f -printf '%TFT%.8TT %p\n' | sort | less (should work on any Single UNIX Specification compliant system) #UNIX #Linux ---- ==> tweet.002.yaml <== status: | When using the find command, use `-iname` instead of `-name` for case-insensitive search #Unix #Linux #Shell #Find ---- ``` ### Split by line number We have already seen how we can use a regular expression to split files. In that case, `csplit` will split the file at the first line **matching* that regex. But you can also identify the split line by its line number as we will see it now. Before switching to YAML, I used to store my scheduled tweets in a [flat file](https://en.wikipedia.org/wiki/Flat%5Ffile%5Fdatabase?ref=linuxhandbook.com). In that file, a tweet was made of two lines. One containing an optional repetition, and the second containing the text of the tweet, with newlines replaced by \\n. Once again [that sample file is available online](https://gist.githubusercontent.com/s-leroux/42377061175b9c323bf5feb2d81bdb65/raw/tweets.txt?ref=linuxhandbook.com). With that “fixed size” format too was able to use `csplit` to put each individual tweet into its own file: ``` sh$ csplit tweets.txt \ > --prefix='tweet.' --suffix-format='%03d.txt' \ > --elide-empty-files \ > --keep-files \ > 2 '{*}' csplit: ‘2’: line number out of range on repetition 62 1 123 222 161 182 119 184 81 148 128 142 101 107 [...] sh$ diff -s tweets.txt <(cat tweet.*.txt) Files tweets.txt and /dev/fd/63 are identical sh$ head tweet.00[012].txt ==> tweet.000.txt <== ==> tweet.001.txt <== { days:180 } I think I use the `sed` command daily. And you?\n\nhttps://www.yesik.it/EP07\n#Shell #Linux #Sed\n#YesIKnowIT ==> tweet.002.txt <== {} Print the first column of a space-separated data file:\nawk '{print $1}' data.txt # Print out just the first column\n\nFor some unknown reason, I find that easier to remember than:\ncut -f1 data.txt\n\n#Linux #AWK #Cut ``` The example above seems easy to understand, but there are two pitfalls here. First, the `2` given as an argument to `csplit` is a line **number*, not a line **count*. However, when using a repetition as I did, after the first match, `csplit` will use that number as a line **count*. If it’s not clear, I let you compare the output of the three following commands: ``` sh$ csplit tweets.txt --keep-files 2 2 2 2 2 csplit: warning: line number ‘2’ is the same as preceding line number csplit: warning: line number ‘2’ is the same as preceding line number csplit: warning: line number ‘2’ is the same as preceding line number csplit: warning: line number ‘2’ is the same as preceding line number 1 0 0 0 0 9030 ``` ``` sh$ csplit tweets.txt --keep-files 2 4 6 8 10 1 123 222 161 182 8342 ``` ``` sh$ csplit tweets.txt --keep-files 2 '{4}' 1 123 222 161 182 8342 ``` I mentioned a second pitfall, somewhat related to the first one. Maybe did you notice the empty line at the very top of the `tweets.txt` file? It leads to that `tweet.000.txt` chunk that contains only the newline character. Unfortunately, it was required in that example because of the repetition: remember I want **two lines* chunks. So the `2` is mandatory before the repetition. But that also means the **first* chunk will break at, but **not including*, the line two. In other words, the first chunk contains one line. All the other ones will contain 2 lines. Maybe you could share your opinion in the comment section, but as of myself I think this was an unfortunate design choice. You can mitigate that issue by skipping directly to the first non-empty line: ``` sh$ csplit tweets.txt \ > --prefix='tweet.' --suffix-format='%03d.txt' \ > --elide-empty-files \ > --keep-files \ > %.% 2 '{*}' csplit: ‘2’: line number out of range on repetition 62 123 222 161 [...] sh$ head tweet.00[012].txt ==> tweet.000.txt <== { days:180 } I think I use the `sed` command daily. And you?\n\nhttps://www.yesik.it/EP07\n#Shell #Linux #Sed\n#YesIKnowIT ==> tweet.001.txt <== {} Print the first column of a space-separated data file:\nawk '{print $1}' data.txt # Print out just the first column\n\nFor some unknown reason, I find that easier to remember than:\ncut -f1 data.txt\n\n#Linux #AWK #Cut ==> tweet.002.txt <== {} For the #shell #beginners :\n« #GlobPatterns : how to move hundreds of files in not time [1/3] »\nhttps://youtu.be/TvW8DiEmTcQ\n\n#Unix #Linux\n#YesIKnowIT ``` ### Reading from stdin Of course, like most of the command line tools, `csplit` can read the input data from its standard input. In that case, you have to specify `-` as the input filename: ``` sh$ tr [:lower:] [:upper:] < tweets.txt | csplit - \ > --prefix='tweet.' --suffix-format='%03d.txt' \ > --elide-empty-files \ > --keep-files \ > %.% 2 '{3}' 123 222 161 8524 sh$ head tweet.???.txt ==> tweet.000.txt <== { DAYS:180 } I THINK I USE THE `SED` COMMAND DAILY. AND YOU?\N\NHTTPS://WWW.YESIK.IT/EP07\N#SHELL #LINUX #SED\N#YESIKNOWIT ==> tweet.001.txt <== {} PRINT THE FIRST COLUMN OF A SPACE-SEPARATED DATA FILE:\NAWK '{PRINT $1}' DATA.TXT # PRINT OUT JUST THE FIRST COLUMN\N\NFOR SOME UNKNOWN REASON, I FIND THAT EASIER TO REMEMBER THAN:\NCUT -F1 DATA.TXT\N\N#LINUX #AWK #CUT ==> tweet.002.txt <== {} FOR THE #SHELL #BEGINNERS :\N« #GLOBPATTERNS : HOW TO MOVE HUNDREDS OF FILES IN NOT TIME [1/3] »\NHTTPS://YOUTU.BE/TVW8DIEMTCQ\N\N#UNIX #LINUX\N#YESIKNOWIT ==> tweet.003.txt <== {} WANT TO KNOW THE OLDEST FILE IN YOUR DISK?\N\NFIND / -TYPE F -PRINTF '%TFT%.8TT %P\N' | SORT | LESS\N(SHOULD WORK ON ANY SINGLE UNIX SPECIFICATION COMPLIANT SYSTEM)\N#UNIX #LINUX {} WHEN USING THE FIND COMMAND, USE `-INAME` INSTEAD OF `-NAME` FOR CASE-INSENSITIVE SEARCH\N#UNIX #LINUX #SHELL #FIND {} FROM A POSIX SHELL `$OLDPWD` HOLDS THE NAME OF THE PREVIOUS WORKING DIRECTORY:\NCD /TMP\NECHO YOU ARE HERE: $PWD\NECHO YOU WERE HERE: $OLDPWD\NCD $OLDPWD\N\N#UNIX #LINUX #SHELL #CD {} FROM A POSIX SHELL, "CD" IS A SHORTHAND FOR CD $HOME\N#UNIX #LINUX #SHELL #CD {} HOW TO MOVE HUNDREDS OF FILES IN NO TIME?\NUSING THE FIND COMMAND!\N\NHTTPS://YOUTU.BE/ZMEFXJYZAQK\N#UNIX #LINUX #MOVE #FILES #FIND\N#YESIKNOWIT ``` And that’s pretty all I wanted to show you today. I hope in the future, you’ll use csplit to split files in Linux. If you’ve enjoyed this article and don’t forget to share and like it on your favorite social network!