08 - OS & Network Basics
Core OS Concepts
Relevant DSS-P Skills
- 3. Technology > 3.1 Software Development > Cloud Infrastructure Utilization
- System call - The programmatic way in which a computer program requests a service from the kernel of the operating system on which it is executed
- Protection ring - A mechanism to protect data and functionality from faults and malicious behavior
- Daemon - A computer program that runs as a background process, rather than being under the direct control of an interactive user
- Environment variable - A named variable whose value is set outside the program, typically through functionality built into the operating system or a microservice
- POSIX standard - A family of standards specified by the IEEE Computer Society for maintaining compatibility between operating systems
Process Management
- Process - The instance of a computer program that is being executed by one or more threads
- Thread - The smallest sequence of programmed instructions that can be managed independently by a scheduler
- Scheduling - The action of assigning resources to perform tasks
- Context switch - The process of storing the state of a process or thread, so that it can be restored and resume execution at a later point
- Interrupt - A request for the processor to interrupt currently executing code, so that the event can be processed in a timely manner
Inter-Process Communication (IPC)
- Pipes
- Anonymous pipe - A simplex FIFO communication channel that may be used for one-way interprocess communication
- Named pipe - An extension to the traditional pipe concept on Unix and Unix-like systems, and is one of the methods of inter-process communication
- Shared memory - A memory that may be simultaneously accessed by multiple programs with an intent to provide communication among them or avoid redundant copies
- Signal - An asynchronous notification sent to a process or to a specific thread within the same process in order to notify it of an event that occurred
- Unix domain socket - A data communications endpoint for exchanging data between processes executing on the same host operating system
Memory Management
- Virtual memory - A memory management technique that provides an idealized abstraction of the storage resources that are actually available on a given machine
- Memory paging - A memory management scheme by which a computer stores and retrieves data from secondary storage for use in main memory
- Page fault - A type of exception raised by computer hardware when a running program accesses a memory page that is not currently mapped by the memory management unit into the virtual address space of a process
- Resident set size (RSS) - The portion of memory occupied by a process that is held in main memory
- Working set size (WSS) - The set of pages in the virtual address space of the process that are currently resident in main memory
- Page cache - A hardware or software component that stores data so that future requests for that data can be served faster
Storage Management
- Disk partitioning - The creation of one or more regions on a secondary storage device, so that each region can be managed separately
- Loop device - A pseudo-device that makes a file accessible as a block device
- File system - A method and data structure that the operating system uses to control how data is stored and retrieved
- Journaling file system - A file system that keeps a journal, a circular log of changes that have not yet been committed to the main part of the file system
- Path - The general form of the name of a file or directory, specifies a unique location in a file system
- Glob pattern - A pattern that specifies sets of filenames with wildcard characters
- File handle/descriptor - A unique identifier for a file or other input/output resource, such as a pipe or network socket
- Symbolic link - A term for any file that contains a reference to another file or directory in the form of an absolute or relative path and that affects pathname resolution
- Permissions - A feature of many modern file systems which control the ability of the users of a computer to view, change, navigate, and execute the contents of the file system
- Setuid - A Unix access rights flag that allows users to run an executable with the permissions of the executable's owner or group
- Sticky bit - A user ownership access right flag that can be assigned to files and directories on Unix-like systems
- Inode - A data structure in a Unix-style file system that describes a file-system object such as a file or a directory
- RAID - A data storage virtualization technology that combines multiple physical disk drive components into one or more logical units for the purposes of data redundancy, performance improvement, or both
Base Network Concepts & Protocols
Relevant DSS-P Skills
- 3. Technology > 3.1 Software Development > Cloud Infrastructure Utilization
- The OSI Model - A conceptual model that provides a common basis for the coordination of standards development for the purpose of systems interconnection
Link Layer (L2)
- Ethernet - A family of wired computer networking technologies
- ARP - A communication protocol used for discovering the link layer address, such as a MAC address, associated with a given internet layer address
- MAC address - A unique identifier assigned to a network interface controller for use as a network address in communications within a network segment
- VLAN - A broadcast domain that is partitioned and isolated in a computer network at the data link layer
- ARP - A communication protocol used for discovering the link layer address, such as a MAC address, associated with a given internet layer address
Internet Layer (L3)
- The Internet - The global system of interconnected computer networks that uses the Internet protocol suite to communicate between networks and devices
- IP - The network layer communications protocol in the Internet protocol suite
- Link-local address - A network address that is valid only for communications within the network segment or the broadcast domain that the host is connected to
- IP-multicast - A method of sending Internet Protocol datagrams to a group of interested receivers in a single transmission
- IPv6 - The most recent version of the Internet Protocol, the communications protocol that provides an identification and location system for computers on networks and routes traffic across the Internet
- Unique local address - An IPv6 address in the address block fc00::/7
- ICMP - A supporting protocol in the Internet protocol suite
- ICMPv6 - The implementation of the Internet Control Message Protocol for Internet Protocol version 6
- DHCP - A network management protocol used on Internet Protocol networks for automatically assigning IP addresses and other communication parameters to devices connected to the network
- DHCPv6 - A network protocol for configuring Internet Protocol version 6 hosts with IP addresses, IP prefixes and other configuration data required to operate in an IPv6 network
- NAT - A method of mapping an IP address space into another by modifying network address information in the IP header of packets while they are in transit across a traffic routing device
- NAT64 - An IPv6 transition mechanism that facilitates communication between IPv6 and IPv4 hosts
- NDP - A protocol in the Internet protocol suite used with Internet Protocol Version 6
- Routing
- Routing table - A data table stored in a router or a network host that lists the routes to particular network destinations
- CIDR - A method for allocating IP addresses and for IP routing
Transport Layer (L4)
- Network socket - A software structure within a network node of a computer network that serves as an endpoint for sending and receiving data across the network
- TCP - A main protocol of the Internet protocol suite
- TCP window scale option - An option to increase the receive window size allowed in Transmission Control Protocol above its former maximum value of 65,535 bytes
- UDP - A core member of the Internet protocol suite
- QUIC - A UDP-based, stream-multiplexing, encrypted transport protocol
Network Architectures
- Peer-to-peer - A distributed computing or networking architecture in which participants share resources directly without reliance on a centralized administrative system
Domain Name System (DNS)
Relevant DSS-P Skills
- 3. Technology > 3.1 Software Development > Cloud Infrastructure Utilization
- DNS - The hierarchical and decentralized naming system used to identify computers, services, and other resources reachable through the Internet or other Internet Protocol networks
- mDNS - A protocol that resolves hostnames to IP addresses within small networks that do not include a local name server
Domain Registration & Lookup
- IANA WHOIS Service - A service to look up the registration data of a domain name or IP address
- Registration Data Access Protocol (RDAP) - A computer network communications protocol that delivers registration data from Domain Name Registries and Regional Internet Registries
Server & Resolver Implementations
- BIND (dnsutils) - A very flexible, full-featured DNS system
- dnsmasq - A lightweight, easy to configure DNS forwarder, DHCP and router advertisement server
- CoreDNS - A DNS server that chains plugins
- systemd-resolved - A system service that provides network name resolution to local applications
- mDNS Implementations
Client Tools
-
Part of BIND
-
dog - A command-line DNS client
-
Doggo - A modern command-line DNS client (like dig) written in Go
-
Managed DNS Services
- Amazon Route53 - A highly available and scalable cloud Domain Name System web service
- Google Cloud DNS - A high-performance, resilient, global Domain Name System service that publishes your domain names to the global DNS in a cost-effective way
Email System
Relevant DSS-P Skills
- 3. Technology > 3.1 Software Development > Cloud Infrastructure Utilization
- Email - A method of exchanging messages between people using electronic devices
- SMTP - A communication protocol for electronic mail transmission
- POP - An application-layer Internet standard protocol used by e-mail clients to retrieve e-mail from a mail server
- IMAP - An Internet standard protocol used by email clients to retrieve email messages from a mail server over a TCP/IP connection
- MIME - A standard that extends the format of email messages to support text in character sets other than ASCII
- Quoted-printable encoding - An encoding that represents data in the 8-bit ASCII character set, so that it can be sent using a 7-bit data path
- Base64 - A group of binary-to-text encoding schemes that represent binary data in an ASCII string format by translating it into a radix-64 representation
Mailbox Formats
- Unix Mbox
- Maildir
Server Software (MTA/MDA)
- Postfix - A mail server that started life at IBM research as an alternative to the widely-used Sendmail program
- Maddy Mail Server - An all-in-one mail server that implements all functionality required to run a mail service
- IMAP
- Cyrus IMAP - A highly scalable enterprise mail system designed for use in small to large enterprise environments
- Dovecot - An open source IMAP and POP3 email server for Linux/UNIX-like systems
Client Software & Utilities
- TUI Clients & Utilities
- mailutils - A set of libraries and utilities for handling electronic mail
- mail command - A command to send and receive mail
- Himalaya - A CLI application for email management, built in Rust, that allows users to interact with emails through shell commands, with support for IMAP, Maildir, SMTP, OAuth 2.0, and PGP encryption
- Mutt - A small but very powerful text based program for reading and sending electronic mail under unix operating systems
- swaks - A featureful, flexible, scriptable, transaction-oriented SMTP test tool
- Pop - A library for sending emails from your terminal
- GNU sharutils - A set of utilities for creating and unpacking shell archives
- mailutils - A set of libraries and utilities for handling electronic mail
- Libraries
- go-mail - A simple to use, yet feature rich mail library for Go
- GUI Clients
- Thunderbird - A free email application that’s easy to set up and customize
- Sylpheed - A simple, lightweight but featureful, and easy-to-use e-mail client
Spam Test and Reputation
-
mail-tester - A free online service that allows you to test your emails for Spam, Malformed Content and Mail Server Configuration problems
-
Spamhaus Project - A non-profit organization that tracks spam and related cyber threats
-
Cloud Services
- Amazon SES - A cost-effective, flexible, and scalable email service that enables developers to send mail from within any application
- SendGrid - A cloud-based email delivery platform that provides reliable transactional and marketing email delivery at scale
Unix-like Operating Systems
Relevant DSS-P Skills
- 3. Technology > 3.1 Software Development > Cloud Infrastructure Utilization
- The Linux Kernel - The main component of a Linux operating system and is the core interface between a computer's hardware and its processes
- Threads
- Pthreads - An execution model that exists independently from a programming language, as well as a parallel execution model
- Filesystems
- Ext4 - The default file system for many major Linux distributions
- XFS - A high-performance journaling file system created by Silicon Graphics, Inc
- Btrfs - A copy-on-write filesystem for Linux with advanced features including snapshots, RAID, and self-healing
- UnionFS - A filesystem service for Linux, FreeBSD and NetBSD which implements a union mount for other file systems
- OverlayFS - A union mount filesystem implementation for Linux
- proc.5 - A virtual filesystem that provides an interface to kernel data structures
- sysfs.5 - A virtual filesystem that exports information about various kernel subsystems, hardware devices, and associated device drivers
- Container Support
- cgroups - A Linux kernel feature which allow processes to be organized into hierarchical groups whose usage of various types of resources can then be limited and monitored
- namespaces - A wrapper for a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource
- lxc/rootfs - The userspace interface for the Linux kernel containment features
- nsenter - A command that executes a program in the namespaces of other processes
- FUSE (Filesystem in Userspace) - An interface for userspace programs to export a filesystem to the Linux kernel
- s3fs - A FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem
- eBPF (Extended Berkeley Packet Filter) - A revolutionary technology with origins in the Linux kernel that can run sandboxed programs in a privileged context
- Threads
Linux Distributions
- General Purpose
- Arch Linux - A simple, lightweight distribution
- Debian - A complete Free Operating System
- Fedora - An innovative, free and open-source operating system platform built by the community for hardware, clouds, and containers
- Gentoo - A highly flexible, source-based Linux distribution that emphasizes customization and performance
- NixOS - A Linux distribution with a unique approach to package and configuration management
- openSUSE - A free Linux operating system for desktops, servers and containers
- Server-focused
- Ubuntu server - The standard platform for public clouds, on-premises, and IoT devices
- Desktop-focused
- Debian-based
- Ubuntu desktop - The Linux-based operating system that runs from the desktop, to the cloud, to all your internet connected things
- BunsenLabs Linux - A distribution offering a light-weight and easily customizable Openbox desktop
- Arch-based
- Manjaro Linux - A user-friendly Linux distribution based on the independently developed Arch operating system
- Mabox Linux - Fast, lightweight and functional Linux Desktop "relaxed" rolling-release, Manjaro based with Openbox Window Manager
- Debian-based
BSD Distributions
- FreeBSD - A free and open-source Unix-like operating system descended from the Berkeley Software Distribution (BSD)
- NetBSD - A free, fast, secure, and highly portable Unix-like Open Source operating system
- OpenBSD - A free and open-source, Unix-like operating system as a fork of NetBSD that emphasizes extremely high security and cryptography
System Services & Auth
- Systemd - A system and service manager for Linux operating systems
- journald - A system service that collects and stores logging data
- hostnamed - A system service that may be used to control the hostname and related machine metadata from user programs
- networkd - A system service that manages networks
- resolved - A system service that provides network name resolution to local applications
- timesyncd - A system service that may be used to synchronize the local system clock with a remote Network Time Protocol server
- linux-pam - A system of libraries that handle the authentication tasks of applications and services in a Linux system
Machine Virtualization
Relevant DSS-P Skills
- 3. Technology > 3.1 Software Development > Cloud Infrastructure Utilization
- Virtualization - The act of creating a virtual version of something, including virtual computer hardware platforms, storage devices, and computer network resources
- libvirt - A toolkit to manage virtualization platforms
Type-1 Hypervisors
- KVM - A full virtualization solution for Linux on x86 hardware containing virtualization extensions
- Hyper-V - A hardware virtualization product from Microsoft
- Proxmox VE - A complete, open-source server management platform for enterprise virtualization
Type-2 Hypervisors
- VirtualBox - A powerful x86 and AMD64/Intel64 virtualization product for enterprise as well as home use
- QEMU - A generic and open source machine emulator and virtualizer
CPU Emulators
- QEMU - A generic and open source machine emulator and virtualizer
Computer Hardware
- CPU Architectures
- CPU Extensions
Linux Host Administration
Relevant DSS-P Skills
- 3. Technology > 3.1 Software Development > Cloud Infrastructure Utilization
Core Utilities
- util-linux - A random collection of Linux utilities
- rsync - An open source utility that provides fast incremental file transfer
- sudo - A system administrator to delegate authority to give certain users the ability to run some commands as root or another user
- shadow-utils - includes the necessary programs for converting UNIX password files to the shadow password format, plus programs for managing user and group accounts
- useradd - A low level utility for adding users
- jc - A CLI tool and Python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries
- aha - An Ansi HTML Adapter
- NO_COLOR - An environment variable to disable ANSI color in command-line software
- Vixie Cron - An open source implementation of POSIX Cron
- Crontab.guru - A quick and simple editor for cron schedule expressions
- logrotate - allows for the automatic rotation compression, removal and mailing of log files
- Syslog - A standard for message logging
Process & System Monitoring
- procps - A set of command line and full-screen utilities that provide information out of the pseudo-filesystem most commonly located at /proc
- ps - A command that displays information about a selection of the active processes
- top - A program that provides a dynamic real-time view of a running system
- free - A command that displays the total amount of free and used physical and swap memory in the system
- vmstat - A command that reports information about processes, memory, paging, block IO, traps, disks and cpu activity
- psmisc - A package of small utilities that use the proc file-system
- lsof - A command for LiSting Open Files
- strace - A diagnostic, debugging and instructional userspace utility for Linux
- inxi - A full featured system information script
- witr - A tool that explains the causal ancestry and purpose of running processes
- Performance Monitors
- sysstat - A collection of performance monitoring tools for Linux
- iostat - A command used for monitoring system input/output device loading
- Monit - A small Open Source utility for managing and monitoring Unix systems
- atop - An ASCII full-screen performance monitor for Linux
- smem - A tool that can give numerous reports on memory usage on Linux systems
- sysstat - A collection of performance monitoring tools for Linux
Time Synchronization
- NTP - A networking protocol for clock synchronization between computer systems over packet-switched, variable-latency data networks
- chrony - A versatile implementation of the Network Time Protocol
- pool.ntp.org - A big virtual cluster of timeservers providing reliable easy to use NTP service for millions of clients
Modern CLI Alternatives
- gdu - A fast disk usage analyzer with a console interface written in Go
- lsd - A rewrite of GNU ls with a lot of added features like colors, icons, tree-view, and more formatting options
- eza - A modern replacement for ls
- broot - A new way to see and navigate directory trees
- bat - A cat(1) clone with wings
- dust - A more intuitive version of du in rust
- dua - A tool to view disk space usage and delete unwanted data, fast
- duf - A better 'df' alternative
- procs - A modern replacement for ps written in Rust
- htop - An interactive process viewer for Unix systems
- btop++ - A resource monitor for Linux, macOS, and FreeBSD
- glances - A cross-platform monitoring tool which aims to present a large amount of monitoring information through a curses or Web based interface
- neofetch - A command-line system information tool
Package Management Tools
- dpkg - The base package management system for Debian
- apt - A command-line utility for installing, updating, removing, and otherwise managing deb packages on Ubuntu, Debian, and related Linux distributions
- Pacman - A utility which manages software packages in Linux
- Yay - An AUR Helper Written in Go
- yum - An automatic updater and package installer/remover for rpm systems
- dnf - The next-generation version of yum
- Homebrew - The Missing Package Manager for macOS (or Linux)
- pipx - A tool to install and run Python applications in isolated environments
- Flatpak - A system for building, distributing, and running sandboxed desktop applications on Linux
- Snapcraft - A software packaging and deployment system developed by Canonical for operating systems that use the Linux kernel
- arkade - A portable marketplace for downloading your favourite devops CLIs and installing helm charts to your Kubernetes cluster
Linux Network Administration
Relevant DSS-P Skills
- 3. Technology > 3.1 Software Development > Cloud Infrastructure Utilization
Configuration & Management
- iproute2 - A collection of utilities for controlling TCP / IP networking and traffic control in Linux
- net-tools (legacy) - A collection of programs for controlling the network subsystem of the Linux kernel
- NetworkManager - A daemon that sits on top of libudev and other Linux kernel interfaces and provides a high-level interface for network configuration
- Ubuntu NetPlan - A network configuration abstraction renderer
Analysis & Security
- tcpdump - A powerful command-line packet analyzer
- wireshark - The world's foremost network protocol analyzer
- nmap - An open source tool for network exploration and security auditing
- ncat - A feature-packed networking utility which reads and writes data across networks from the command line
- traceroute - A computer network diagnostic tool for displaying the route and measuring transit delays of packets across an Internet Protocol network
Proxies & Gateways
- SOCKS Proxy - An Internet protocol that exchanges network packets between a client and server through a proxy server
- Dante - A SOCKS server and SOCKS client, implementing RFC 1928 and related standards
- tun2socks - A SOCKS proxy for TCP and UDP, that handles all connections from a TUN device
- proxychains - A tool that forces any TCP connection made by any given application to follow through proxy like TOR or any other SOCKS4, SOCKS5 or HTTP(S) proxy
File Sharing & Remote Access
Relevant DSS-P Skills
- 3. Technology > 3.1 Software Development > Cloud Infrastructure Utilization
File Servers and Protocols
- SMB - A network communication protocol for providing shared access to files, printers, and serial ports between nodes on a network
- Samba - The standard Windows interoperability suite of programs for Linux and Unix
- FTP - A standard communication protocol used for the transfer of computer files from a server to a client on a computer network
- vsftpd - A GPL licensed FTP server for UNIX-like systems, including Linux
- SFTP - A network protocol that provides file access, file transfer, and file management over any reliable data stream
- SFTPGo - A fully featured and highly configurable SFTP server with optional HTTP/S, FTP/S and WebDAV support
Remote Access Servers and Protocols
- SSH - A cryptographic network protocol for operating network services securely over an unsecured network
- openssh - The premier connectivity tool for remote login with the SSH protocol
- RDP - A proprietary protocol developed by Microsoft which provides a user with a graphical interface to connect to another computer over a network connection
- xrdp - An open-source Remote Desktop Protocol server
- RFB - A simple protocol for remote access to graphical user interfaces
- Mosh - A replacement for interactive SSH terminals