Introduction
I asked my colleagues what tasks they had to do in their careers which taught them a valuable lesson. This page lists a series of their responses plus some of my own as well as relavent links and explanations. The tasks generally get harder as you go down the list.
Table of Contents
Package an RPM
Configure a LAMPs stack from scratch
Explore the Cisco CLI
Install an Arch Linux Desktop
Buy a domain and run your own authoritative DNS servers
Practice root password recovery
Use Ansible to Automate Configuration Management
Set up a replicated Postgres SQL cluster with encrypted communication
Setup an Infiniband fabric
Build a Linux kernel from scratch
Setup an OpenVPN server
Benchmarking with HPL
Setup a provisioning server
Configure LDAP
Configure Kerberos
Migrate an installed system
Build Linux From Scratch
Package an RPM
Packaging software is an important skill and the RPM is found in lots of places. Find your favorite open source project with a spec file and build a custom package.
Objectives:
- Learn how package management works
- Create a package and install it
Setup:
- Machine running CentOS
Links:
Follow on activity:
- Learn about the Debian Reproducible Builds project
Configure a LAMPs stack from scratch
Standing up a web service is a very common task and it is important to know the general stack: from web server to database. Wordpress is a good application to begin with.
Objectives:
- Learn about:
- Installing software
- Configuring daemons using text files
- SQL databases
- Firewalls
- Networking
- Practice installing a common building block for many applications
Setup:
- Virtual machine and a web browser
Links:
Follow on activity:
- Replicate using FreeBSD
Explore the Cisco CLI
Understanding the OSI model and how network components work together is very important. If you have a medium-sized corporate network laying around, then you are probably all set. Otherwise, you might look at Ebay for old gear or look into setting up an emulated network lab.
A basic task for many admins is tracing a client’s connection back to a physical switch port.
Objectives:
- Become familiar with Cisco’s CLI
- Learn about layer 2 and layer 3 networking
Setup:
- SSH access to several inter-connected Cisco devices
Links:
- Cisco TAC Training - CLI Basics
- Network Lessons Cisco Introduction
- Cisco Forums - Tracing Hosts
- Stack Exchange
Follow on activity:
- Set up a virtual GNS3 network lab to configure:
- Routers
- Switches
- Firewalls
- GNS3
Install an Arch Linux Desktop
Arch is a popular distribution with a great community behind it. Using this distribution day in and out introduces you to the latest and greatest open source software, and it does not hide the complexity of the system from you. Another great distribution for immersion learning is Gentoo.
Objectives:
- Learn about:
- Partitions and File Systems
- Boot loaders
- Installing Linux step by step
- Networking
- Experience a different take on Linux and OS distribution
Setup:
- Blank virtual machine
- Internet access
Links:
Follow on activity:
Buy a domain and run your own authoritative DNS servers
It’s not DNS
There’s no way it’s DNS
It was DNS
-Kind Internet Fellow
Objectives:
- Learn about:
- Role of registrars
- Domain Name System
- BIND
- Practice setting up forward, reverse, and resolving name servers
- Manage zones by hand using using reversion control
Setup:
- Digitial Ocean/Linode/Vultr/EC2/GCE instance
- Credit card
Links:
- DigitalOcean - Authoritative DNS server guide
- MicroHowto - Configure BIND as a slave
- DigitalOcean - BIND as a caching and forwarding server
- IntoDNS DNS Checker
Follow on activity:
- Configure DNSSEC for your domain
Practice root password recovery
Take a modern Linux distribution which protects single user mode and reset the root password. Bonus points if you don’t use boot media.
Objectives:
- Learn about:
- Boot loaders
- Kernel booting
- Init process
- Single user mode
- Perform password recovery
Setup:
- Centos 7 virtual machine
- Console access
Links:
Use Ansible to Automate Configuration Management
Automating system tasks allows an admin to amplify their energy, and using a framework allows for better organization and reusability. Focus on learning one particular system well but also take away a general understanding so that you can transition quickly to a new framework.
Objectives:
- Learn about:
- Ansible configuration management
- Source code management
Setup:
- Two or three virtual machines and/or cloud instances
Links:
- RedHat - Getting started with Ansible Fast
- Linode - Learn How to Run Ansible
- Servers for Hackers
- Scotch.io - Getting Started with Ansible
- Github - Try Git
- Ansible - The Essential Best Practices
- Ansible - Best Practices
- Ansible - Style Guide
Set up a replicated Postgres SQL cluster with encrypted communication
Single instance MySQL databases are ubiquitous but PostgreSQL is a popular choice for niche and high performance usecases. Replicated PostgreSQL is an easy path to a HA SQL setup.
Objectives:
- Learn about:
- Postgres SQL and types of high availability
- Encryption
- Postgres authentication and authorization
- Practice setting up a active/read-only replicated cluster
Setup:
- Two virtual machines
Links:
Follow on activity:
- Learn about NoSQL databases for system monitoring
Setup an Infiniband fabric
Infiniband is a departure from the world of ubiquituous Ethernet. It is used anywhere latency is critical: supercomputing, software defined storage, message queues, and database systems. A key benefit is cutting out IP and TCP moving to RDMA communication between subcomponents.
Ebay is a good source of cheap 10Gbps (SDR) or 20Gbps (DDR) parts at reasonable prices. Expect to invest several hundred dollars though.
Objectives:
- Learn about:
- Setting up Infiniband networking
- Benchmark latencies and bandwidth
- Install and configure OFED and MPI
Setup:
- A simple setup needs only two computers, Infiniband cards, and one cable
- Adding an Infiniband switch creates a complete fabric
Links:
- Wikipedia - Infiniband
- Mellanox - Introduction to Infiniband PDF
- SC16 Tutorial - Infiniband for Dummies
- ArchLinux Wiki - Infiniband
- Ohio State University - MVAPICH Benchmarks
- Qemu - RDMA Live Migration
Build a Linux kernel from scratch
The heart of a computer is the kernel and it is an adventure to work directly with Linux.
Objectives:
- Learn about the Linux kernel
- Practice knowledge of boot loaders and the init process
Setup:
- Virtual machine running Linux
- Internet access
- Copy of the latest kernel.org kernel or LTS kernel
Links:
- Linux
- TLDP - Building a Custom Kernel
- Linode - Custom Compiled Knerel on Debian and Ubuntu
- Debian Handbook - Compiling a Kernel
- Ubuntu Wiki - Build your own kernel
- Centos Wiki - Custom Kernel
- ArchLinux Kernels
Setup an OpenVPN server
VPNs are incredibly useful and OpenVPN is very popular. Setting up a VPS with OpenVPN properly is a great exercise and an excuse to practice with OpenSSL, NAT, routing, and more.
Objectives:
- Learn about:
- Networking
- VPNs
- Routing
- Firewalls
- RSA/SSL
Setup:
- Virtual machine with a public IP
- Client with the OpenVPN client and Internet access
Links:
Follow on activity:
- Explore Wireguard
Benchmarking with HPL
HPL is the High Performance Linpack benchark used by the Top500 list of fastest supercomputers in the world. It has been used for more than a decade to rank systems and it quantifies the performance of your processor.
Compiling HPL requires an installation of high performance compiler, MPI library for communication, and linear algebra library for math functions.
Objective:
- Learn about:
- System benchmarking
- Basic research libraries
- Application compiling
- Development toolchain
Setup:
- Linux server or Linux cluster
Links:
- ArchLinux - Stress testing
- Top500 - The Linpack Benchmark
- Netlib - HPL
- MPICH - Guides
- Slothparadise - How to compile HPL
Setup a provisioning server
Having a push button installation procedure saves incredible amounts of time. There are many ways to accomplish automated OS installs from custom scripts to distribution specific tools (Kickstart or Preseed or FAI) to generic frameworks like Cobbler.
Objectives:
- Learn about:
- Classic daemons
- DNS
- DHCP
- HTTP(S)
- TFTP
- Network booting a server
- Classic daemons
Setup:
- Two VM’s
Links:
- Redhat RHEL7 Install Guide
- RedHat RHEL5 Install Guide
- Building a fully automated Ubuntu Installation Process
- Debian Preseed
- Debian Fully Autoatmic Installation (FAI)
- Cobbler
- IBM - Automate and manage systems installation with Cobbler
- DigitalOcean - Configure BIND
- Prefetch Technologies - Installing, Configuring, and Debugging the ISC DHCP Server
- UbuntuGeek - Setup Advanced TFTP server
- PXELINUX
- iPXE
- iPXE - Chainloading iPXE
Configure LDAP
There are many variations and uses for LDAP. A common case is to put shared user account information in OpenLDAP. Configuring a directory and Linux clients is a good first step, but many situations call for replication of the database and encryption.
Objectives:
- Learn about:
- OpenLDAP
- LDAP management
- Linux authentication and authorization
- Practice sharing user accounts between servers
Setup:
- Handful of virtual machines
Links:
- DigitalOcean - How to Install and Configure OpenLDAP
- OpenLDAP - Quick-Start Guide
- TLDP - LDAP Linux HOWTO
- HowtoForge - OpenLDAP Installation and Configuration
Configure Kerberos
Kerberos is a hugely popular authentication service (thanks Microsoft…?) Many sites that rely on single sign-on will have a Kerberos domain somewhere in their enterprise.
You will learn the value of properly set clocks.
Objectives:
- Learn about:
- Kerberos
- Strong authentication systems
- Practice setting up a realm with SSH authentication and NFS storage
Setup:
- Several virtual machines
Links:
Follow on activity:
- Explore NTP and networked time sources
Migrate an installed system
Migrating into a VM, changing disk layouts, and restoring from backups are some of the many uses for migrating a root file system by file.
Objectives:
- Learn about restoring to bare metal
- Practice a combination of skills
Setup:
- An installed virtual machine
- A blank virtual machine
- Boot media
Links:
- ArchLinux Wiki - Moving an existing install
- Anchor - Migrating server to a virtual machine
- Linux Journal - Virtualizing a server with minimal downtime
Build Linux From Scratch
Objectives:
- Learn about the GNU in GNU/Linux
- Practice wizardry
Links:
Follow on activity:
- Study operating systems, starting with “Operating System Concepts” by Silbershatz, Galvin, and Cagne