Skip to main content

08 - OS & Network Basics

Core OS Concepts​

  • System call - The programmatic way in which a computer program requests a service from the kernel of the operating system on which it is executed
  • Protection ring - A mechanism to protect data and functionality from faults and malicious behavior
  • Daemon - A computer program that runs as a background process, rather than being under the direct control of an interactive user
  • Environment variable - A named variable whose value is set outside the program, typically through functionality built into the operating system or a microservice
  • POSIX standard - A family of standards specified by the IEEE Computer Society for maintaining compatibility between operating systems

Process Management​

  • Process - The instance of a computer program that is being executed by one or more threads
    • Thread - The smallest sequence of programmed instructions that can be managed independently by a scheduler
    • Scheduling - The action of assigning resources to perform tasks
    • Context switch - The process of storing the state of a process or thread, so that it can be restored and resume execution at a later point
    • Interrupt - A request for the processor to interrupt currently executing code, so that the event can be processed in a timely manner

Inter-Process Communication (IPC)​

  • Pipes
    • Anonymous pipe - A simplex FIFO communication channel that may be used for one-way interprocess communication
    • Named pipe - An extension to the traditional pipe concept on Unix and Unix-like systems, and is one of the methods of inter-process communication
  • Shared memory - A memory that may be simultaneously accessed by multiple programs with an intent to provide communication among them or avoid redundant copies
  • Signal - An asynchronous notification sent to a process or to a specific thread within the same process in order to notify it of an event that occurred
  • Unix domain socket - A data communications endpoint for exchanging data between processes executing on the same host operating system

Memory Management​

  • Virtual memory - A memory management technique that provides an idealized abstraction of the storage resources that are actually available on a given machine
    • Memory paging - A memory management scheme by which a computer stores and retrieves data from secondary storage for use in main memory
    • Page fault - A type of exception raised by computer hardware when a running program accesses a memory page that is not currently mapped by the memory management unit into the virtual address space of a process
    • Resident set size (RSS) - The portion of memory occupied by a process that is held in main memory
    • Working set size (WSS) - The set of pages in the virtual address space of the process that are currently resident in main memory
  • Page cache - A hardware or software component that stores data so that future requests for that data can be served faster

Storage Management​

  • Disk partitioning - The creation of one or more regions on a secondary storage device, so that each region can be managed separately
  • Loop device - A pseudo-device that makes a file accessible as a block device
  • File system - A method and data structure that the operating system uses to control how data is stored and retrieved
    • Journaling file system - A file system that keeps a journal, a circular log of changes that have not yet been committed to the main part of the file system
    • Path - The general form of the name of a file or directory, specifies a unique location in a file system
    • Glob pattern - A pattern that specifies sets of filenames with wildcard characters
    • File handle/descriptor - A unique identifier for a file or other input/output resource, such as a pipe or network socket
    • Symbolic link - A term for any file that contains a reference to another file or directory in the form of an absolute or relative path and that affects pathname resolution
    • Permissions - A feature of many modern file systems which control the ability of the users of a computer to view, change, navigate, and execute the contents of the file system
      • Setuid - A Unix access rights flag that allows users to run an executable with the permissions of the executable's owner or group
      • Sticky bit - A user ownership access right flag that can be assigned to files and directories on Unix-like systems
    • Inode - A data structure in a Unix-style file system that describes a file-system object such as a file or a directory
  • RAID - A data storage virtualization technology that combines multiple physical disk drive components into one or more logical units for the purposes of data redundancy, performance improvement, or both

Base Network Concepts & Protocols​

  • The OSI Model - A conceptual model that provides a common basis for the coordination of standards development for the purpose of systems interconnection
  • Ethernet - A family of wired computer networking technologies
    • ARP - A communication protocol used for discovering the link layer address, such as a MAC address, associated with a given internet layer address
      • MAC address - A unique identifier assigned to a network interface controller for use as a network address in communications within a network segment
    • VLAN - A broadcast domain that is partitioned and isolated in a computer network at the data link layer

Internet Layer (L3)​

  • The Internet - The global system of interconnected computer networks that uses the Internet protocol suite to communicate between networks and devices
  • IP - The network layer communications protocol in the Internet protocol suite
    • Link-local address - A network address that is valid only for communications within the network segment or the broadcast domain that the host is connected to
    • IP-multicast - A method of sending Internet Protocol datagrams to a group of interested receivers in a single transmission
    • IPv6 - The most recent version of the Internet Protocol, the communications protocol that provides an identification and location system for computers on networks and routes traffic across the Internet
    • ICMP - A supporting protocol in the Internet protocol suite
    • ICMPv6 - The implementation of the Internet Control Message Protocol for Internet Protocol version 6
    • DHCP - A network management protocol used on Internet Protocol networks for automatically assigning IP addresses and other communication parameters to devices connected to the network
    • DHCPv6 - A network protocol for configuring Internet Protocol version 6 hosts with IP addresses, IP prefixes and other configuration data required to operate in an IPv6 network
    • NAT - A method of mapping an IP address space into another by modifying network address information in the IP header of packets while they are in transit across a traffic routing device
    • NAT64 - An IPv6 transition mechanism that facilitates communication between IPv6 and IPv4 hosts
    • NDP - A protocol in the Internet protocol suite used with Internet Protocol Version 6
  • Routing
    • Routing table - A data table stored in a router or a network host that lists the routes to particular network destinations
    • CIDR - A method for allocating IP addresses and for IP routing

Transport Layer (L4)​

  • Network socket - A software structure within a network node of a computer network that serves as an endpoint for sending and receiving data across the network
  • TCP - A main protocol of the Internet protocol suite
    • TCP window scale option - An option to increase the receive window size allowed in Transmission Control Protocol above its former maximum value of 65,535 bytes
  • UDP - A core member of the Internet protocol suite
  • QUIC - A UDP-based, stream-multiplexing, encrypted transport protocol

Domain Name System (DNS)​

  • DNS - The hierarchical and decentralized naming system used to identify computers, services, and other resources reachable through the Internet or other Internet Protocol networks
  • mDNS - A protocol that resolves hostnames to IP addresses within small networks that do not include a local name server

Domain Registration & Lookup​

Server & Resolver Implementations​

  • BIND (dnsutils) - A very flexible, full-featured DNS system
  • dnsmasq - A lightweight, easy to configure DNS forwarder, DHCP and router advertisement server
  • CoreDNS - A DNS server that chains plugins
  • systemd-resolved - A system service that provides network name resolution to local applications
  • mDNS Implementations
    • Avahi - A system which facilitates service discovery on a local network via the mDNS/DNS-SD protocol suite
    • Bonjour - Apple's implementation of zero-configuration networking

Client Tools​

  • Part of BIND
    • dig - A flexible tool for interrogating DNS name servers
    • nslookup - A program to query Internet domain name servers
  • dog - A command-line DNS client
  • Doggo - A modern command-line DNS client (like dig) written in Go

Cloud Services​

  • Amazon Route53 - A highly available and scalable cloud Domain Name System web service
  • Google Cloud DNS - A high-performance, resilient, global Domain Name System service that publishes your domain names to the global DNS in a cost-effective way

Email System​

  • Email - A method of exchanging messages between people using electronic devices
  • SMTP - A communication protocol for electronic mail transmission
  • POP - An application-layer Internet standard protocol used by e-mail clients to retrieve e-mail from a mail server
  • IMAP - An Internet standard protocol used by email clients to retrieve email messages from a mail server over a TCP/IP connection
  • MIME - A standard that extends the format of email messages to support text in character sets other than ASCII
    • Quoted-printable encoding - An encoding that represents data in the 8-bit ASCII character set, so that it can be sent using a 7-bit data path
    • Base64 - A group of binary-to-text encoding schemes that represent binary data in an ASCII string format by translating it into a radix-64 representation

Mailbox Formats​

  • Unix Mbox
  • Maildir

Server Software (MTA/MDA)​

  • Postfix - A mail server that started life at IBM research as an alternative to the widely-used Sendmail program
  • Maddy Mail Server - An all-in-one mail server that implements all functionality required to run a mail service
  • IMAP
    • Cyrus IMAP - A highly scalable enterprise mail system designed for use in small to large enterprise environments
    • Dovecot - An open source IMAP and POP3 email server for Linux/UNIX-like systems

Client Software & Utilities​

  • TUI Clients & Utilities
    • mailutils - A set of libraries and utilities for handling electronic mail
    • Mutt - A small but very powerful text based program for reading and sending electronic mail under unix operating systems
    • swaks - A featureful, flexible, scriptable, transaction-oriented SMTP test tool
    • Pop - A library for sending emails from your terminal
    • GNU sharutils - A set of utilities for creating and unpacking shell archives
  • Libraries
    • go-mail - A simple to use, yet feature rich mail library for Go
  • GUI Clients
    • Thunderbird - A free email application that’s easy to set up and customize
    • Sylpheed - A simple, lightweight but featureful, and easy-to-use e-mail client

Spam Test and Reputation​

  • mail-tester - A free online service that allows you to test your emails for Spam, Malformed Content and Mail Server Configuration problems

  • Spamhaus Project - A non-profit organization that tracks spam and related cyber threats

  • Cloud Services

    • Amazon SES - A cost-effective, flexible, and scalable email service that enables developers to send mail from within any application
    • Twilio SendGrid - A cloud-based email delivery service that helps businesses with email delivery

Linux System Architecture​

  • The Linux Kernel - The main component of a Linux operating system and is the core interface between a computer's hardware and its processes
    • Threads
      • Pthreads - An execution model that exists independently from a programming language, as well as a parallel execution model
    • Filesystems
      • Ext4 - The default file system for many major Linux distributions
      • XFS - A high-performance journaling file system created by Silicon Graphics, Inc
      • UnionFS - A filesystem service for Linux, FreeBSD and NetBSD which implements a union mount for other file systems
      • OverlayFS - A union mount filesystem implementation for Linux
      • proc.5 - A virtual filesystem that provides an interface to kernel data structures
      • sysfs.5 - A virtual filesystem that exports information about various kernel subsystems, hardware devices, and associated device drivers
    • Container Support
      • cgroups - A Linux kernel feature which allow processes to be organized into hierarchical groups whose usage of various types of resources can then be limited and monitored
      • namespaces - A wrapper for a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource
      • lxc/rootfs - The userspace interface for the Linux kernel containment features
      • nsenter - A command that executes a program in the namespaces of other processes
    • FUSE (Filesystem in Userspace) - An interface for userspace programs to export a filesystem to the Linux kernel
      • s3fs - A FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem
    • eBPF (Extended Berkeley Packet Filter) - A revolutionary technology with origins in the Linux kernel that can run sandboxed programs in a privileged context

Linux Distributions​

  • Ubuntu server - The standard platform for public clouds, on-premises, and IoT devices
  • Debian - A complete Free Operating System
  • Arch Linux - A simple, lightweight distribution
  • NixOS - A Linux distribution with a unique approach to package and configuration management

System Services & Auth​

  • Systemd - A system and service manager for Linux operating systems
    • journald - A system service that collects and stores logging data
    • hostnamed - A system service that may be used to control the hostname and related machine metadata from user programs
    • networkd - A system service that manages networks
    • resolved - A system service that provides network name resolution to local applications
    • timesyncd - A system service that may be used to synchronize the local system clock with a remote Network Time Protocol server
  • linux-pam - A system of libraries that handle the authentication tasks of applications and services in a Linux system

Machine Virtualization​

  • Virtualization - The act of creating a virtual version of something, including virtual computer hardware platforms, storage devices, and computer network resources

Type-1 Hypervisors​

  • KVM - A full virtualization solution for Linux on x86 hardware containing virtualization extensions
  • Hyper-V - A hardware virtualization product from Microsoft
  • Proxmox VE - A complete, open-source server management platform for enterprise virtualization

Type-2 Hypervisors​

  • VirtualBox - A powerful x86 and AMD64/Intel64 virtualization product for enterprise as well as home use
  • QEMU - A generic and open source machine emulator and virtualizer

Virtualization Management​

  • libvert - A toolkit to manage virtualization platforms
  • OpenNebula - The Enterprise Cloud and Virtualization Platform that delivers end-to-end coverage, vendor neutrality, and comprehensive support across cloud and virtualization management, hypervisor operations, and Kubernetes orchestration

CPU Emulators​

  • QEMU - A generic and open source machine emulator and virtualizer

Computer Hardware​

Linux Host Administration​

Core Utilities​

  • util-linux - A random collection of Linux utilities
    • lsblk - A command that lists information about all available or the specified block devices
    • lsns - A command that lists information about all the currently accessible namespaces or about the given namespace
    • swapon - A command used to specify devices on which paging and swapping are to take place
  • rsync - An open source utility that provides fast incremental file transfer
  • sudo - A system administrator to delegate authority to give certain users the ability to run some commands as root or another user
  • shadow-utils - includes the necessary programs for converting UNIX password files to the shadow password format, plus programs for managing user and group accounts
    • useradd - A low level utility for adding users
  • jc - A CLI tool and Python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries
  • aha - An Ansi HTML Adapter
    • NO_COLOR - An environment variable to disable ANSI color in command-line software
  • Vixie Cron - An open source implementation of POSIX Cron
    • Crontab.guru - A quick and simple editor for cron schedule expressions
  • logrotate - allows for the automatic rotation compression, removal and mailing of log files
  • Syslog - A standard for message logging

Process & System Monitoring​

  • procps - A set of command line and full-screen utilities that provide information out of the pseudo-filesystem most commonly located at /proc
    • ps - A command that displays information about a selection of the active processes
    • top - A program that provides a dynamic real-time view of a running system
    • free - A command that displays the total amount of free and used physical and swap memory in the system
    • vmstat - A command that reports information about processes, memory, paging, block IO, traps, disks and cpu activity
  • psmisc - A package of small utilities that use the proc file-system
    • pstree - A command that shows running processes as a tree
    • killall - A command that sends a signal to all processes running any of the specified commands
  • lsof - A command for LiSting Open Files
  • strace - A diagnostic, debugging and instructional userspace utility for Linux
  • inxi - A full featured system information script
  • witr - A tool that explains the causal ancestry and purpose of running processes
  • Performance Monitors
    • sysstat - A collection of performance monitoring tools for Linux
      • iostat - A command used for monitoring system input/output device loading
    • Monit - A small Open Source utility for managing and monitoring Unix systems
    • atop - An ASCII full-screen performance monitor for Linux
    • smem - A tool that can give numerous reports on memory usage on Linux systems

Time Synchronization​

  • NTP - A networking protocol for clock synchronization between computer systems over packet-switched, variable-latency data networks
  • chrony - A versatile implementation of the Network Time Protocol
  • pool.ntp.org - A big virtual cluster of timeservers providing reliable easy to use NTP service for millions of clients

Modern CLI Alternatives​

  • lsd - A rewrite of GNU ls with a lot of added features like colors, icons, tree-view, and more formatting options
  • eza - A modern replacement for ls
  • broot - A new way to see and navigate directory trees
  • bat - A cat(1) clone with wings
  • dust - A more intuitive version of du in rust
  • dua - A tool to view disk space usage and delete unwanted data, fast
  • duf - A better 'df' alternative
  • procs - A modern replacement for ps written in Rust
  • htop - An interactive process viewer for Unix systems
  • btop++ - A resource monitor for Linux, macOS, and FreeBSD
  • glances - A cross-platform monitoring tool which aims to present a large amount of monitoring information through a curses or Web based interface
  • neofetch - A command-line system information tool

Package Management Tools​

  • dpkg - The base package management system for Debian
    • apt - A command-line utility for installing, updating, removing, and otherwise managing deb packages on Ubuntu, Debian, and related Linux distributions
  • Pacman - A utility which manages software packages in Linux
    • Yay - An AUR Helper Written in Go
  • yum - An automatic updater and package installer/remover for rpm systems
  • dnf - The next-generation version of yum
  • Homebrew - The Missing Package Manager for macOS (or Linux)
  • pipx - A tool to install and run Python applications in isolated environments
  • Flatpak - A system for building, distributing, and running sandboxed desktop applications on Linux
  • Snapcraft - A software packaging and deployment system developed by Canonical for operating systems that use the Linux kernel
  • arkade - A portable marketplace for downloading your favourite devops CLIs and installing helm charts to your Kubernetes cluster

Linux Network Administration​

Configuration & Management​

  • iproute2 - A collection of utilities for controlling TCP / IP networking and traffic control in Linux
    • ip - The main command to show / manipulate routing, network devices, interfaces and tunnels
    • ss - A utility to investigate sockets
  • net-tools (legacy) - A collection of programs for controlling the network subsystem of the Linux kernel
    • ifconfig - A command used to configure a network interface
    • netstat - A command that prints network connections, routing tables, interface statistics, masquerade connections, and multicast memberships
  • NetworkManager - A daemon that sits on top of libudev and other Linux kernel interfaces and provides a high-level interface for network configuration
  • Ubuntu NetPlan - A network configuration abstraction renderer

Analysis & Security​

  • tcpdump - A powerful command-line packet analyzer
  • wireshark - The world's foremost network protocol analyzer
  • nmap - An open source tool for network exploration and security auditing
    • ncat - A feature-packed networking utility which reads and writes data across networks from the command line
  • traceroute - A computer network diagnostic tool for displaying the route and measuring transit delays of packets across an Internet Protocol network

Proxies & Gateways​

  • SOCKS Proxy - An Internet protocol that exchanges network packets between a client and server through a proxy server
    • Dante - A SOCKS server and SOCKS client, implementing RFC 1928 and related standards
    • tun2socks - A SOCKS proxy for TCP and UDP, that handles all connections from a TUN device
    • proxychains - A tool that forces any TCP connection made by any given application to follow through proxy like TOR or any other SOCKS4, SOCKS5 or HTTP(S) proxy

File Sharing & Remote Access​

File Servers and Protocols​

  • SMB - A network communication protocol for providing shared access to files, printers, and serial ports between nodes on a network
    • Samba - The standard Windows interoperability suite of programs for Linux and Unix
  • FTP - A standard communication protocol used for the transfer of computer files from a server to a client on a computer network
    • vsftpd - A GPL licensed FTP server for UNIX-like systems, including Linux
  • SFTP - A network protocol that provides file access, file transfer, and file management over any reliable data stream
    • SFTPGo - A fully featured and highly configurable SFTP server with optional HTTP/S, FTP/S and WebDAV support

Remote Access Servers and Protocols​

  • SSH - A cryptographic network protocol for operating network services securely over an unsecured network
    • openssh - The premier connectivity tool for remote login with the SSH protocol
  • RDP - A proprietary protocol developed by Microsoft which provides a user with a graphical interface to connect to another computer over a network connection
    • xrdp - An open-source Remote Desktop Protocol server
  • RFB - A simple protocol for remote access to graphical user interfaces
    • x11vnc - A VNC server for X11
    • TightVNC - A free remote desktop application
  • Mosh - A replacement for interactive SSH terminals

Performance & Load Testing​

  • Performance Testing - The practice of evaluating how a system performs in terms of responsiveness and stability under a particular workload

Performance Testing Tools​

  • Hyperfine - A command-line benchmarking tool
  • Locust - An easy-to-use, distributed, user load testing tool
  • Grafana k6 - The open-source load testing tool that makes performance testing easy and productive for engineering teams
  • Gatling - The load testing tool for programmers that helps engineering teams shift performance concerns left
  • Apache Jmeter - A pure Java application designed to load test functional behavior and measure performance
  • ab - A tool for benchmarking your Apache Hypertext Transfer Protocol (HTTP) server
  • stress-ng - A tool that imposes configurable amounts of CPU, memory, I/O, and disk stress on the system
  • sysbench - A scriptable multi-threaded benchmark tool based on LuaJIT
  • fio - A tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user
  • iPerf - The ultimate speed test tool for TCP, UDP and SCTP
  • plow - A high-performance HTTP benchmarking tool