Linux 101 #03: Linux Workflow (Cont)

File Descriptors and Redirections

A file descriptor (FD) in Unix/Linux operating systems is a reference, maintained by the kernel, that allows the system to manage Input/Output (I/O) operations. It acts as a unique identifier for an open file, socket, or any other I/O resource. In Windows-based operating systems, this is known as a file handle. Essentially, the file descriptor is the system’s way of keeping track of active I/O connections, such as reading from or writing to a file.

This knowledge allows us to manipulate how input and output flows between files, processes, and the system, enabling us to handle data more effectively. By leveraging these tools, we can streamline tasks, avoid unnecessary steps, and work with files and system resources in a much more organized and efficient manner, ultimately enhancing our productivity and precision in managing operations.

By default, the first three file descriptors in Linux are:

Data Stream for Input: STDIN – 0
Data Stream for Output: STDOUT – 1
Data Stream for Output that relates to an error occurring: STDERR – 2

Let us see an example with cat. When running cat to read a file, we not used STDIN – FD 0 in this example (unless you input it manually from the keyboard), and it is returned to the terminal as standard output (STDOUT – FD 1) the content of the file creds.

In the next example, by using the find command, we will see the standard output (STDOUT – FD 1) marked in green and standard error (STDERR – FD 2) marked in red.

In this case, the error is marked and displayed with “Permission denied”. We can check this by redirecting the file descriptor for the errors (FD 2 – STDERR) to “/dev/null.” This way, we redirect the resulting errors to the “null device,” which discards all data.

In the next example, by using the find command, we will see the standard output (STDOUT – FD 1) marked in green and standard error (STDERR – FD 2) marked in red.

Now we can see that all errors (STDERR) previously presented with “Permission denied” are no longer displayed. The only result we see now is the standard output (STDOUT), which we can also redirect to a file with the name results.txt that will only contain standard output without the standard errors.

We should have noticed that we did not use a number before the greater-than sign (>) in the last example. That is because we redirected all the standard errors to the “null device” before, and the only output we get is the standard output (FD 1 – STDOUT). To make this more precise, we will redirect standard error (FD 2 – STDERR) and standard output (FD 1 – STDOUT) to different files.

When we use the greater-than sign (>) to redirect our STDOUT, a new file is automatically created if it does not already exist. If this file exists, it will be overwritten without asking for confirmation. If we want to append STDOUT to our existing file, we can use the double greater-than sign (>>).

Another way to redirect STDOUT is to use pipes (|). These are useful when we want to use the STDOUT from one program to be processed by another. One of the most commonly used tools is grep, which we will use in the next example. Grep is used to filter STDOUT according to the pattern we define. In the next example, we use the find command to search for all files in the “/etc/” directory with a “.conf” extension. Any errors are redirected to the “null device” (/dev/null). Using grep, we filter out the results and specify that only the lines containing the pattern “systemd” should be displayed.

The redirections work, not only once. We can use the obtained results to redirect them to another program. For the next example, we will use the tool called wc, which should count the total number of obtained results.

Filter Contents

Now, let’s talk about reading files directly from the command line, without needing to open a text editor. There are two powerful tools for this – more and less. These are known as pagers, and they allow you to view the contents of a file interactively, one screen at a time. While both tools serve a similar purpose, they have some differences in functionality, which we’ll touch on later.

Using more and less, you can easily scroll through large files, search for text, and navigate forward or backward without modifying the file itself. This is especially useful when you’re working with large logs or text files that don’t fit neatly into one screen.

Before we start filtering the output of commands, let’s explore a few foundational tools that will help you efficiently sift through and manipulate text. These tools are crucial when working with large amounts of data or when you need to automate tasks that involve searching, sorting, or processing information. Let’s look at some examples to understand how these tools work.

In this example I try to read the password.list file, it’s contains a thousand lines of password: cat password.list | more.

After we read the content using cat and redirected it to more, the already mentioned pager opens, and we will automatically start at the beginning of the file. With the [Q] key, we can leave this pager. We will notice that the output remains in the terminal.

If we now take a look at the tool less, we will notice on the man page that it contains many more features than more. The presentation is almost the same as with more. When closing less with the [Q] key, we will notice that the output we have seen, unlike more, does not remain in the terminal.

Sometimes we will only be interested in specific issues either at the beginning of the file or the end. If we only want to get the first lines of the file, we can use the tool head. By default, head prints the first ten lines of the given file or input, if not specified otherwise.

If we only want to see the last parts of a file or results, we can use the counterpart of head called tail, which returns the last ten lines.

Depending on which results and files are dealt with, they are rarely sorted. Often it is necessary to sort the desired results alphabetically or numerically to get a better overview. For this, we can use a tool called sort.

In many cases, we will need to search for specific results that match patterns we define. One of the most commonly used tools for this purpose is grep, which provides a wide range of powerful features for pattern searching. For instance, we can use grep to search for iloveyou password in the password list.

This is just one example of how grep can be applied to efficiently filter data based on predefined patterns. Another possibility is to exclude specific results. For this, the option “-v” is used with grep. In the next example, we exclude all users who have disabled the standard shell with the name “/bin/false” or “/usr/bin/nologin”.

Specific results with different characters may be separated as delimiters. Here it is handy to know how to remove specific delimiters and show the words on a line in a specified position. One of the tools that can be used for this is cut. Therefore we use the option “-d” and set the delimiter to the colon character (:) and define with the option “-f” the position in the line we want to output.

Another possibility to replace certain characters from a line with characters defined by us is the tool tr. As the first option, we define which character we want to replace, and as a second option, we define the character we want to replace it with. In the next example, we replace the colon character with space.

As we may have noticed, the line for the user “postgres” has one column too many. To keep it as simple as possible to sort out such results, the (g)awk programming is beneficial, which allows us to display the first ($1) and last ($NF) result of the line.

There will come moments when we want to change specific names in the whole file or standard input. One of the tools we can use for this is the stream editor called sed. One of the most common uses of this is substituting text. Here, sed looks for patterns we have defined in the form of regular expressions (regex) and replaces them with another pattern that we have also defined. Let’s stick to the last results and say I want to replace the word “bin” with “ZED.”

The “s” flag at the beginning stands for the substitute command. Then we specify the pattern we want to replace. After the slash (/), we enter the pattern we want to use as a replacement in the third position. Finally, we use the “g” flag, which stands for replacing all matches.

Permission Management

In Linux, permissions are like keys that control access to files and directories. These permissions are assigned to both users and groups, much like keys being distributed to specific individuals and teams within an organization. Each user can belong to multiple groups, and being part of a group grants additional access rights, allowing users to perform specific actions on files and directories.

Every file and directory has an owner (a user) and is associated with a group. The permissions for these files are defined for both the owner and the group, determining what actions—like reading, writing, or executing—are allowed. When you create a new file or directory, it automatically becomes “yours” and is associated with the group you belong to, similar to how a project within a company might default to your team’s oversight.

The whole permission system on Linux systems is based on the octal number system, and basically, there are three different types of permissions a file or directory can be assigned:

(r) – Read
(w) – Write
(x) – Execute

The permissions can be set for the owner, group, and others like presented in the next example with their corresponding permissions.

We can modify permissions using the chmod command, permission group references (u – owner, g – Group, o – others, a – All users), and either a [+] or a [-] to add or remove the designated permissions. In the following example, let’s assume I have a file called test01.txt and I want to change permissions for it so this script is owned by that user, becomes not executable, and set with write permissions for all users.

We can also set the permissions for all other users to read only using the octal value assignment: chmod 754 test01.txt

Let us look at all the representations associated with it to understand better how the permission assignment is calculated.

If we sum the set bits from the Binary Representation assigned to the values from Binary Notation together, we get the Octal Value. The Permission Representation represents the bits set in the Binary Representation by using the three characters, which only recognizes the set permissions easier.

To change the owner and/or the group assignments of a file or directory, we can use the chown command.

The syntax is like following: chown <user>:<group> <file/directory>

In addition to standard user and group permissions, Linux allows us to configure special permissions on files through the Set User ID (SUID) and Set Group ID (SGID) bits. These bits function like temporary access passes, enabling users to run certain programs with the privileges of another user or group. For example, administrators can use SUID or SGID to grant users elevated rights for specific applications, allowing tasks to be performed with the necessary permissions, even if the user themselves doesn’t normally have them.

The presence of these permissions is indicated by an s in place of the usual x in the file’s permission set. When a program with the SUID or SGID bit set is executed, it runs with the permissions of the file’s owner or group, rather than the user who launched it. This can be useful for certain system tasks but also introduces potential security risks if not used carefully.

One common risk is when administrators, unfamiliar with an application’s full functionality, assign SUID or SGID bits indiscriminately. And here’s a pretty useful trick after you’ve compromised a target and need to escalate privileges to another user — or even to root — if the sysadmin has misconfigured programs with SUID or SGID bits. You can read more on GTFOBins.

Last but not least, I’ll talk about Sticky bits. Sticky bits in Linux are like locks on files within shared spaces. When set on a directory, the sticky bit adds an extra layer of security, ensuring that only certain individuals can modify or delete files, even if others have access to the directory.

This feature is especially useful in shared environments, like public directories, where multiple users are working together. By setting the sticky bit, you ensure that important files aren’t accidentally or maliciously altered by someone who shouldn’t have the authority to do so, adding an important safeguard to collaborative workspaces.

Alright, I think that’s enough for the Linux Workflow section. It’d be really hard to cover everything about Linux in these blog posts anyway. If you truly want to learn Linux, the best way is just to install it on your machine using VMware or VirtualBox — then dive in and start practicing.