Skip to main content

08 - OS & Network Basics

Core OS Concepts​

  • Core Concepts
    • System call - The programmatic way in which a computer program requests a service from the kernel of the operating system on which it is executed
    • Protection ring - A mechanism to protect data and functionality from faults and malicious behavior
    • Daemon - A computer program that runs as a background process, rather than being under the direct control of an interactive user
    • Environment variable - A named variable whose value is set outside the program, typically through functionality built into the operating system or a microservice
    • POSIX standard - A family of standards specified by the IEEE Computer Society for maintaining compatibility between operating systems
  • Process Management
    • Process - The instance of a computer program that is being executed by one or more threads
      • Thread - The smallest sequence of programmed instructions that can be managed independently by a scheduler
      • Scheduling - The action of assigning resources to perform tasks
      • Context switch - The process of storing the state of a process or thread, so that it can be restored and resume execution at a later point
      • Interrupt - A request for the processor to interrupt currently executing code, so that the event can be processed in a timely manner
  • Inter-Process Communication (IPC)
    • Pipes
      • Anonymous pipe - A simplex FIFO communication channel that may be used for one-way interprocess communication
      • Named pipe - An extension to the traditional pipe concept on Unix and Unix-like systems, and is one of the methods of inter-process communication
    • Shared memory - A memory that may be simultaneously accessed by multiple programs with an intent to provide communication among them or avoid redundant copies
    • Signal - An asynchronous notification sent to a process or to a specific thread within the same process in order to notify it of an event that occurred
    • Unix domain socket - A data communications endpoint for exchanging data between processes executing on the same host operating system
  • Memory Management
    • Virtual memory - A memory management technique that provides an idealized abstraction of the storage resources that are actually available on a given machine
      • Memory paging - A memory management scheme by which a computer stores and retrieves data from secondary storage for use in main memory
      • Page fault - A type of exception raised by computer hardware when a running program accesses a memory page that is not currently mapped by the memory management unit into the virtual address space of a process
      • Resident set size (RSS) - The portion of memory occupied by a process that is held in main memory
      • Working set size (WSS) - The set of pages in the virtual address space of the process that are currently resident in main memory
    • Page cache - A hardware or software component that stores data so that future requests for that data can be served faster
  • Storage Management
    • Disk partitioning - The creation of one or more regions on a secondary storage device, so that each region can be managed separately
    • Loop device - A pseudo-device that makes a file accessible as a block device
    • File system - A method and data structure that the operating system uses to control how data is stored and retrieved
      • Journaling file system - A file system that keeps a journal, a circular log of changes that have not yet been committed to the main part of the file system
      • Path - The general form of the name of a file or directory, specifies a unique location in a file system
      • Glob pattern - A pattern that specifies sets of filenames with wildcard characters
      • File handle/descriptor - A unique identifier for a file or other input/output resource, such as a pipe or network socket
      • Symbolic link - A term for any file that contains a reference to another file or directory in the form of an absolute or relative path and that affects pathname resolution
      • Permissions - A feature of many modern file systems which control the ability of the users of a computer to view, change, navigate, and execute the contents of the file system
        • Setuid - A Unix access rights flag that allows users to run an executable with the permissions of the executable's owner or group
        • Sticky bit - A user ownership access right flag that can be assigned to files and directories on Unix-like systems
      • Inode - A data structure in a Unix-style file system that describes a file-system object such as a file or a directory
    • RAID - A data storage virtualization technology that combines multiple physical disk drive components into one or more logical units for the purposes of data redundancy, performance improvement, or both

Base Network Concepts & Protocols​

Note: Please see also 220 - Domain Name System and Email, 300 - Web and API Style Standards, and 630 - PKI and Secure Communications

  • The OSI Model - A conceptual model that provides a common basis for the coordination of standards development for the purpose of systems interconnection
  • The Internet - The global system of interconnected computer networks that uses the Internet protocol suite to communicate between networks and devices
    • IP - The network layer communications protocol in the Internet protocol suite
      • Link-local address - A network address that is valid only for communications within the network segment or the broadcast domain that the host is connected to
      • IP-multicast - A method of sending Internet Protocol datagrams to a group of interested receivers in a single transmission
      • DHCP - A network management protocol used on Internet Protocol networks for automatically assigning IP addresses and other communication parameters to devices connected to the network
      • ICMP - A supporting protocol in the Internet protocol suite
      • NAT - A method of mapping an IP address space into another by modifying network address information in the IP header of packets while they are in transit across a traffic routing device
      • IPv6 - The most recent version of the Internet Protocol, the communications protocol that provides an identification and location system for computers on networks and routes traffic across the Internet
        • Unique local address - An IPv6 address in the address block fc00::/7
        • DHCPv6 - A network protocol for configuring Internet Protocol version 6 hosts with IP addresses, IP prefixes and other configuration data required to operate in an IPv6 network
        • ICMPv6 - The implementation of the Internet Control Message Protocol for Internet Protocol version 6
        • NAT64 - An IPv6 transition mechanism that facilitates communication between IPv6 and IPv4 hosts
        • NDP - A protocol in the Internet protocol suite used with Internet Protocol Version 6
    • Routing table - A data table stored in a router or a network host that lists the routes to particular network destinations
      • CIDR - A method for allocating IP addresses and for IP routing
    • Network socket - A software structure within a network node of a computer network that serves as an endpoint for sending and receiving data across the network
    • TCP - A main protocol of the Internet protocol suite
      • TCP window scale option - An option to increase the receive window size allowed in Transmission Control Protocol above its former maximum value of 65,535 bytes
    • UDP - A core member of the Internet protocol suite
    • QUIC - A UDP-based, stream-multiplexing, encrypted transport protocol
  • Ethernet - A family of wired computer networking technologies
    • ARP - A communication protocol used for discovering the link layer address, such as a MAC address, associated with a given internet layer address
      • MAC address - A unique identifier assigned to a network interface controller for use as a network address in communications within a network segment
    • VLAN - A broadcast domain that is partitioned and isolated in a computer network at the data link layer

Domain Name System (DNS)​

  • Core Concepts & Protocols
    • DNS - The hierarchical and decentralized naming system used to identify computers, services, and other resources reachable through the Internet or other Internet Protocol networks
    • mDNS - A protocol that resolves hostnames to IP addresses within small networks that do not include a local name server
  • Domain Registration & Lookup
  • Server & Resolver Implementations
    • BIND (dnsutils) - A very flexible, full-featured DNS system
    • dnsmasq - A lightweight, easy to configure DNS forwarder, DHCP and router advertisement server
    • CoreDNS - A DNS server that chains plugins
    • systemd-resolved - A system service that provides network name resolution to local applications
    • mDNS Implementations
      • Avahi - A system which facilitates service discovery on a local network via the mDNS/DNS-SD protocol suite
      • Bonjour - Apple's implementation of zero-configuration networking
  • Client Tools
    • Part of BIND
      • dig - A flexible tool for interrogating DNS name servers
      • nslookup - A program to query Internet domain name servers
    • dog - A command-line DNS client
    • Doggo - A modern command-line DNS client (like dig) written in Go
  • Cloud Services
    • Amazon Route53 - A highly available and scalable cloud Domain Name System web service
    • Google Cloud DNS - A high-performance, resilient, global Domain Name System service that publishes your domain names to the global DNS in a cost-effective way

Email System​

  • Core Concepts & Protocols
    • Email - A method of exchanging messages between people using electronic devices
    • SMTP - A communication protocol for electronic mail transmission
    • POP - An application-layer Internet standard protocol used by e-mail clients to retrieve e-mail from a mail server
    • IMAP - An Internet standard protocol used by email clients to retrieve email messages from a mail server over a TCP/IP connection
    • MIME - A standard that extends the format of email messages to support text in character sets other than ASCII
      • Quoted-printable encoding - An encoding that represents data in the 8-bit ASCII character set, so that it can be sent using a 7-bit data path
      • Base64 - A group of binary-to-text encoding schemes that represent binary data in an ASCII string format by translating it into a radix-64 representation
  • Mailbox Formats
    • Unix Mbox
    • Maildir
  • Server Software (MTA/MDA)
    • Postfix - A mail server that started life at IBM research as an alternative to the widely-used Sendmail program
    • Maddy Mail Server - An all-in-one mail server that implements all functionality required to run a mail service
    • IMAP
      • Cyrus IMAP - A highly scalable enterprise mail system designed for use in small to large enterprise environments
      • Dovecot - An open source IMAP and POP3 email server for Linux/UNIX-like systems
  • Client Software & Utilities
    • TUI Clients & Utilities
      • mailutils - A set of libraries and utilities for handling electronic mail
      • Mutt - A small but very powerful text based program for reading and sending electronic mail under unix operating systems
      • swaks - A featureful, flexible, scriptable, transaction-oriented SMTP test tool
      • Pop - A library for sending emails from your terminal
      • GNU sharutils - A set of utilities for creating and unpacking shell archives
    • Libraries
      • go-mail - A simple to use, yet feature rich mail library for Go
    • GUI Clients
      • Thunderbird - A free email application that’s easy to set up and customize
      • Sylpheed - A simple, lightweight but featureful, and easy-to-use e-mail client
  • Spam Test and Reputation
    • mail-tester - A free online service that allows you to test your emails for Spam, Malformed Content and Mail Server Configuration problems
    • Spamhaus Project - A non-profit organization that tracks spam and related cyber threats
  • Cloud Services
    • Amazon SES - A cost-effective, flexible, and scalable email service that enables developers to send mail from within any application
    • Twilio SendGrid - A cloud-based email delivery service that helps businesses with email delivery

Linux Kernel & Main Components​

Note: Please see also 210 - Linux Administration

  • The Linux Kernel - The main component of a Linux operating system and is the core interface between a computer's hardware and its processes
    • Threads
      • Pthreads - An execution model that exists independently from a programming language, as well as a parallel execution model
    • Filesystems
      • ext4 - The default file system for many major Linux distributions
      • xfs - A high-performance journaling file system created by Silicon Graphics, Inc
      • overlayfs - A union mount filesystem implementation for Linux
      • proc.5 - A virtual filesystem that provides an interface to kernel data structures
      • sysfs.5 - A virtual filesystem that exports information about various kernel subsystems, hardware devices, and associated device drivers
    • Container Support
      • cgroups - A Linux kernel feature which allow processes to be organized into hierarchical groups whose usage of various types of resources can then be limited and monitored
      • namespaces - A wrapper for a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource
      • lxc/rootfs - The userspace interface for the Linux kernel containment features
      • nsenter - A command that executes a program in the namespaces of other processes
    • FUSE (Filesystem in Userspace) - An interface for userspace programs to export a filesystem to the Linux kernel
      • s3fs - A FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem
    • eBPF (Extended Berkeley Packet Filter) - A revolutionary technology with origins in the Linux kernel that can run sandboxed programs in a privileged context
  • Linux distros for hosts
    • Ubuntu server - The standard platform for public clouds, on-premises, and IoT devices
    • Debian - A complete Free Operating System
    • Arch Linux - A simple, lightweight distribution
  • linux-pam - A system of libraries that handle the authentication tasks of applications and services in a Linux system
  • Systemd - A system and service manager for Linux operating systems
    • journald - A system service that collects and stores logging data
    • hostnamed - A system service that may be used to control the hostname and related machine metadata from user programs
    • networkd - A system service that manages networks
    • resolved - A system service that provides network name resolution to local applications
    • timesyncd - A system service that may be used to synchronize the local system clock with a remote Network Time Protocol server

Virtualization​

  • Virtualization - The act of creating a virtual version of something, including virtual computer hardware platforms, storage devices, and computer network resources
  • Type-1 Hypervisors
    • KVM - A full virtualization solution for Linux on x86 hardware containing virtualization extensions
    • Hyper-V - A hardware virtualization product from Microsoft
    • Proxmox VE - A complete, open-source server management platform for enterprise virtualization
  • Type-2 Hypervisors
    • VirtualBox - A powerful x86 and AMD64/Intel64 virtualization product for enterprise as well as home use
    • QEMU - A generic and open source machine emulator and virtualizer
  • Vitrualization Management
    • libvert - A toolkit to manage virtualization platforms
  • CPU Emulators
    • QEMU - A generic and open source machine emulator and virtualizer

Computer Hardware​

Linux Host Administration​

  • Basic Tools
    • util-linux - A random collection of Linux utilities
      • lsblk - A command that lists information about all available or the specified block devices
      • lsns - A command that lists information about all the currently accessible namespaces or about the given namespace
      • swapon - A command used to specify devices on which paging and swapping are to take place
    • rsync - An open source utility that provides fast incremental file transfer
    • Vixie Cron - An open source implementation of POSIX Cron
    • logrotate - allows for the automatic rotation compression, removal and mailing of log files
    • Syslog - A standard for message logging
    • procps - A set of command line and full-screen utilities that provide information out of the pseudo-filesystem most commonly located at /proc
      • ps - A command that displays information about a selection of the active processes
      • top - A program that provides a dynamic real-time view of a running system
      • free - A command that displays the total amount of free and used physical and swap memory in the system
      • vmstat - A command that reports information about processes, memory, paging, block IO, traps, disks and cpu activity
    • psmisc - A package of small utilities that use the proc file-system
      • pstree - A command that shows running processes as a tree
      • killall - A command that sends a signal to all processes running any of the specified commands
    • lsof - A command for LiSting Open Files
    • sudo - A system administrator to delegate authority to give certain users the ability to run some commands as root or another user
    • shadow-utils - includes the necessary programs for converting UNIX password files to the shadow password format, plus programs for managing user and group accounts
      • useradd - A low level utility for adding users
    • strace - A diagnostic, debugging and instructional userspace utility for Linux
    • inxi - A full featured system information script
    • Monitors
      • Monit - A small Open Source utility for managing and monitoring Unix systems
      • atop - An ASCII full-screen performance monitor for Linux
      • sysstat - A collection of performance monitoring tools for Linux
        • iostat - A command used for monitoring system input/output device loading
      • smem - A tool that can give numerous reports on memory usage on Linux systems
    • Clock syncing
      • NTP - A networking protocol for clock synchronization between computer systems over packet-switched, variable-latency data networks
      • chrony - A versatile implementation of the Network Time Protocol
      • pool.ntp.org - A big virtual cluster of timeservers providing reliable easy to use NTP service for millions of clients
    • jc - A CLI tool and Python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries
    • aha - An Ansi HTML Adapter
      • NO_COLOR - An environment variable to disable ANSI color in command-line software
  • Modern Tools
    • lsd - A rewrite of GNU ls with a lot of added features like colors, icons, tree-view, and more formatting options
    • eza - A modern replacement for ls
    • broot - A new way to see and navigate directory trees
    • bat - A cat(1) clone with wings
    • dust - A more intuitive version of du in rust
    • dua - A tool to view disk space usage and delete unwanted data, fast
    • duf - A better 'df' alternative
    • procs - A modern replacement for ps written in Rust
    • htop - An interactive process viewer for Unix systems
    • btop++ - A resource monitor for Linux, macOS, and FreeBSD
    • glances - A cross-platform monitoring tool which aims to present a large amount of monitoring information through a curses or Web based interface
    • neofetch - A command-line system information tool
  • Package Management Tools
    • dpkg - The base package management system for Debian
      • apt - A command-line utility for installing, updating, removing, and otherwise managing deb packages on Ubuntu, Debian, and related Linux distributions
    • Pacman - A utility which manages software packages in Linux
      • Yay - An AUR Helper Written in Go
    • Homebrew - The Missing Package Manager for macOS (or Linux)
    • pipx - A tool to install and run Python applications in isolated environments
    • Flatpak - A system for building, distributing, and running sandboxed desktop applications on Linux
    • Snapcraft - A software packaging and deployment system developed by Canonical for operating systems that use the Linux kernel
    • arkade - A portable marketplace for downloading your favourite devops CLIs and installing helm charts to your Kubernetes cluster

Linux Network Administration​

  • Basic Tools
    • iproute2 - A collection of utilities for controlling TCP / IP networking and traffic control in Linux
      • ip - The main command to show / manipulate routing, network devices, interfaces and tunnels
      • ss - A utility to investigate sockets
    • net-tools (legacy) - A collection of programs for controlling the network subsystem of the Linux kernel
      • ifconfig - A command used to configure a network interface
      • netstat - A command that prints network connections, routing tables, interface statistics, masquerade connections, and multicast memberships
    • traceroute - A computer network diagnostic tool for displaying the route and measuring transit delays of packets across an Internet Protocol network
    • NetworkManager - A daemon that sits on top of libudev and other Linux kernel interfaces and provides a high-level interface for network configuration
    • Ubuntu NetPlan - A network configuration abstraction renderer
    • tcpdump - A powerful command-line packet analyzer
    • wireshark - The world's foremost network protocol analyzer
    • nmap - An open source tool for network exploration and security auditing
      • ncat - A feature-packed networking utility which reads and writes data across networks from the command line
  • Layer 5 Gateway
    • SOCKS Proxy - An Internet protocol that exchanges network packets between a client and server through a proxy server
      • Dante - A SOCKS server and SOCKS client, implementing RFC 1928 and related standards
      • tun2socks - A SOCKS proxy for TCP and UDP, that handles all connections from a TUN device
      • proxychains - A tool that forces any TCP connection made by any given application to follow through proxy like TOR or any other SOCKS4, SOCKS5 or HTTP(S) proxy

File Sharing & Remote Access​

  • File Servers and Protocols
    • SMB - A network communication protocol for providing shared access to files, printers, and serial ports between nodes on a network
      • Samba - The standard Windows interoperability suite of programs for Linux and Unix
    • FTP - A standard communication protocol used for the transfer of computer files from a server to a client on a computer network
      • vsftpd - A GPL licensed FTP server for UNIX-like systems, including Linux
    • SFTP - A network protocol that provides file access, file transfer, and file management over any reliable data stream
      • SFTPGo - A fully featured and highly configurable SFTP server with optional HTTP/S, FTP/S and WebDAV support
  • Remote Access Servers and Protocols
    • SSH - A cryptographic network protocol for operating network services securely over an unsecured network
      • openssh - The premier connectivity tool for remote login with the SSH protocol
    • RDP - A proprietary protocol developed by Microsoft which provides a user with a graphical interface to connect to another computer over a network connection
      • xrdp - An open-source Remote Desktop Protocol server
    • RFB - A simple protocol for remote access to graphical user interfaces
      • x11vnc - A VNC server for X11
      • TightVNC - A free remote desktop application
    • Mosh - A replacement for interactive SSH terminals

Performance & Load Testing​

  • Concepts
    • Performance Testing - The practice of evaluating how a system performs in terms of responsiveness and stability under a particular workload
  • Performance Testing Tools
    • Grafana k6 - The open-source load testing tool that makes performance testing easy and productive for engineering teams
    • Gatling - The load testing tool for programmers that helps engineering teams shift performance concerns left
    • Apache Jmeter - A pure Java application designed to load test functional behavior and measure performance
    • ab - A tool for benchmarking your Apache Hypertext Transfer Protocol (HTTP) server
    • stress-ng - A tool that imposes configurable amounts of CPU, memory, I/O, and disk stress on the system
    • sysbench - A scriptable multi-threaded benchmark tool based on LuaJIT
    • fio - A tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user
    • iPerf - The ultimate speed test tool for TCP, UDP and SCTP
    • plow - A high-performance HTTP benchmarking tool