System Security

System Security is both:

• The prevention of unauthorized access, and operation
• The protection from malicious actors who could be trying to:
  • Steal Data
  • Cause Damage to a System
  • Disrupt Services

SE Linux

Security-enhanced Linux Layer of policies on top of normal OS execution Rules to specify what can / can’t be accessed

User Authentication & Logging

Users are validated by passwords or keys, and their activity gets logged into system log files, which can be inspected, audited and mined. (e.g. dmesg or /var/log/messages)

Access Control

We know how UNIX file permissions work: Unix File Permissions

• R/W/X
• U/G/O and how we can use chmod to set these permission bits

So how can we give things like executables extra permissions?

setuid

sudo is commonly used to perform a command “as the root”

A more nuanced, controlled approach is through the use of the setuid bit, which allows users to run exe with the same permissions as the file’s owner. This allows a temporary privilege elevation for a specific task. This can be done with chmod u+s file.exe

Setting Limits

Occasionally, and especially on larger shared networks, you want to cap user activity to prevent single bad actors from taking up too much of the system resources This is the purpose of the ulimit command, which can set thresholds on certain activities such as:

• Max processes forked
• Max files open
• etc

You can view the current limits on your unix system with ulimit -a

Fork Bomb

This is a process that copies itself recursively, it is also frequently referred to as a “wabbit”

fork() {fork | fork & }; fork, or more simply :(){:|:&};:

This can be seen as a Local DoS attack

Denial Of Service (DOS)

String Sanitization

User-generated strings can be dangerous and must always be sanitized before being used or passed in as parameters String length should also always be bounded Some malicious techniques used with strings are:

• Stack Smashing - Overwriting the return address by going over the limit
• Arbitrary Code Execution - Like SQL Injection

Some possible solutions?

• Address space layout randomization
• pointer authentication codes (PAC)
• CHERI

Hardware-based Exploits

Bit-flipping in memory - e.g. Rowhammer Leaking privileged info - e.g. Meltdown

Rowhammer

This is a RAM vunerability due to optimisation techniques in DDR3 and DDR4 RAM

It gives bad actors a non-zero probability of flipping a bit in a given row, by alternated accesses to adjacent rows.

Meltdown

User can gain read access to privileged kernel memory

Speculative execution attacks exploit the fact that a CPU will already start accessing data before it knows if it is allowed to, i.e., while the memory protection check is in progress This is to do with a concept called branch prediciton where if there is a complex check required, it will use the system memory to predict the most likely branch and then begin executing it before the check is complete, in the case of a prediction miss, it will roll back the progress made

However, the protected data is stored in the cache regardless of the privilege of the process. Cache memory can be accessed more quickly than regular memory. The attacker process can try to access memory locations to test if the data there has been cached, by timing the access.

Verification

Static analysis provides guarantees about safety of OS components - e.g. no infinite loops, no race conditions. (this is how eBPF works)

Verified properties include access control guarantees, memory safety, and system call termination Proofs very complex, difficult to construct

seL4 is a fully verified microkernel system

Certification

Certification ensures the integrity and authenticity of operating system components, verifying that they have not been tampered with or altered maliciously before installation or execution.

1. Checking Integrity of Binary Distributions

• Integrity checks compare the current state of OS binaries against a previously known good state using checksums or cryptographic hashes (e.g., SHA-256).
• Upon downloading or transferring files, their hashes are recalculated and compared to published values to detect any alterations.

2. Guaranteeing Authenticity through Hashes

• Hashes published by software providers allow users to verify their downloaded files against these trusted references, ensuring files are unmodified and as released by the developers.

3. Cryptographic Signatures for Provenance

• Files are signed by developers using private keys, with the corresponding public keys made available for verification.
• These signatures confirm file authenticity and verify the source, ensuring files come from legitimate and trusted sources.

4. Preventing Various Attacks

• Integrity attacks: Modifications like malicious code insertions are detected through mismatched hashes.
• Man-in-the-middle attacks (MITM): Alterations during file transmission are detectable by hash and signature discrepancies.
• Spoofing attacks: Cryptographic signatures help authenticate the source, reducing the risk of installing malicious software impersonating legitimate applications.

User and Kernel Mode

To isolate user code and prevent it from damaging the OS kernel code, user applications are run in ‘user mode’ which is less privileged than ‘kernel mode’

What can’t you do in ‘user mode’? Examples …

• execute privileged instructions - e.g. mode switch instructions, I/O
• access memory that doesn’t belong to this application
• set interrupt flags

Our system boots in kernel mode, but will generally stay in user mode until an interrupt is handled at which point it can switch to kernel mode Syscalls being executed also switch us into kernel mode