Ansible

This post is to help me page ansible back into memory if it has been moved to long term storage.

Overview

an automation tool. Ansible is agentless, meaning:
    - you only have to install and run ansible on the control node.
    - you must have connectivity from ansible to the things you are trying to control (automation targets).
        - *nix -SSH
        - Windows - winrm
        - Cloud and other stuff - API

Ansible is pretty flexible. For any given thing, you can do or describe it several different ways, e.g. describe it on the command line, in a file, in a database that gets read, etc. So, in the example, don't assume any of these things are necessarily set in stone.

Ansible is run from the command line of the control node with either ansible or ansible-playbook. Both of those commands take the option -i <path to inventory files>

Terminology

Playbook

A yaml file describing the desired state of the automation target. The file consists of:

  • hosts - directive specifies an inventory group to which to apply plays
  • variables - These variables for convenience, not parameterization. That is, use them when you don't want to have to dig through the playbook and update things in all the various places they get used, e.g., like a path variable. It's best if you don't use these variables for something you change for each automation target
  • plays - see below
  • handler - see below

Plays

Plays are for organization.

They group tasks (see below). The tasks are executed in order unless the sequence is altered with one of the flow control directives.

Each play is named because helps

  • force commenting
  • lable sections of output/logs
  • organize sections of a playbook
     

Tasks

Tasks generally result in discrete actions in on the automation target.

Tasks are named, for the same reason plays are named.

Tasks call a module, usually in the format

        <module>: <directive 1>=<value> <directive 2>=<value>

Handlers

Handlers are like tasks in that they direct an action. Unlike tasks, handlers:

  • are executed after all other tasks in a playbook
  • are only executed if running the playbook resulted in a change

Handlers are useful for things like restarting a service when a change was made to its configuration.

Inventory

The inventory is how ansible describes which host/automation targets to work on. The inventory can be files or generated dynamically. The main focus of the inventory is to group hosts.

Inventory Files -

While you can specify variables in inventory files, but best practice is to separate variables out into separate files.
Inventory files are the only ansible file than can be a non-yaml format, that is the INI format.

Groups

Groups are a way to manage multiple hosts similarly A group can consist of hosts or of other groups. In an inventory file, a group of groups uses the children keyword. Look up the syntax. Hosts can belong to multiple groups. If they belong to a nested group, they are members of both the parent and the child group. That is the, the parent is a union of children.

When specifying hosts in calls to ansible or ansible-playbook, you can do fun stuff like excluding from groups, the intersection of groups, etc. See patterns.

There are two default groups: all and ungrouped

Variables

Variables are a way to change ansible's behavior for a given task.

Variable Declaration

Variables can be declared in so many different places that the documentation has a long list of precedence. The documentation says it's more important to know where you should declare a variable than to worry about order of precedence. The documentation then goes on to begrudgingly document the order of precedence in excruciating detail and completely omits a description of the best place to declare variables.

Variable names must be letters, numbers, and _. The name must contain at least one letter. note: no hypen (-). Don't use two leading and trailing underscores, like __name__, or you will run afoul of reserved python keywords.

Two major types of variables apply to

  • individual hosts- called host_vars or host variables groups. Usually declared in a file called /etc/ansible/host_vars/hostname.yaml
  • groups- called group_vars or group variables. Usually declared in a file called /etc/ansible/group_vars/groupname.yaml

Variable Reference

When you run ansible against a host or group, it looks up the applicable variables in the host and group variable files, as well as a number of other places. The value is then used when a playbook references the variable using the Jinja2 system. E.g.

        {{variable name}}

Jinja2 has filters which are a way to apply functions to manipulate the value of the variable. There are lots of string and integer manipulation functions.
    
A task can output a value that you can register, i.e. declare, for use by subsequent tasks.

Best practice for organization

By default, all your playbooks, inventories, variables, etc. are rooted in the directory /etc/ansible. I can't see when that would ever be practical, but I guess they had to have some default. Instead, people generally just choose some directory as the root, like say specific to the client they are working on. Then they make all their ansible calls with -i /path/to/that/root.

Within that directory, the structure is as follows. This deviates a bit from what ansible says is the standard because I think making names more explicit is useful:

    hosts-inventory.yaml                # the inventory file. note, this can be .yaml, .yml, no extension.
    group_vars/one_file_per_group.yaml        
    host_vars/one_file_per_host.yaml
    
    site-playbook.yaml                    # master playbook that calls are the child playbooks
    group1-playbook.yaml                # playbook for group 1
    group2-playbook.yaml        

 

git commit Messages

I like to use a git hook to automatically prepend all commit messages with the name of the branch I am working on.

Create the file .git/hooks/prepare-commit-msg with the contents:

#!/bin/bash

branch=$(git symbolic-ref -q --short HEAD) || "<no branch>"
sed -i .bak -e "1s:^:[$branch] :" $1

Make the file executable, and that's all there is to it.

chmod u+x git/hooks/prepare-commit-msg

Setting up an AWS CLI Development Environment

Directories

I maintain each client’s files in a separate directory hierarchy. I may transition to maintaining each in their own VM, but for now, the directory structure is sufficient. Within the client directory, I create an AWS subdirectory. So,

  1. Create client directory.
  2. Create aws subdirectory.

Access Keys

The AWS CLI tools authenticates with an access key ID and a secret access key. You can get the keys from the AWS console. Then, the CLI tools will look in a number of different places for the access keys. The easiest way would probably be to create a profile file, but I feel queasy about saving a password in clear text to a file system. Instead, I transfer the risk to a my password manager. Then, I export them to environment variables whenever I am doing work for that client.

1. Generate a set of keys. (https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html)
2. Store the keys in my password manager. (Don’t download) There, I store them in the format

 export AWS_ACCESS_KEY_ID=AKIA...
 export AWS_SECRET_ACCESS_KEY=xyz...

Note that there is a single space in front of the export. This ensures that when you copy the command and execute it, the key won't end up in the command history.
I also include the following in my .bashrc, but I am a belt and suspenders kind of guy.\

export HISTIGNORE="clear:history:export"

3. Double-check there aren’t any keys stored in other places that will cause conflicts. In fact, I store the following in ~/.aws/credentials just to ensure I’m not accidentally saving anything to disk.

[default]
aws_access_key_id = AKIAdefaultshouldnotworkanywhere
aws_secret_access_key = +defaultshouldnotworkanywhere

CLI Tools

1. Ensure you have the latest AWS CLI tools. I develop on a Mac, so I use pip.

sudo -H pip3 install --upgrade awscli

2. Then you can test you have everything working correctly by running a simple describe command.

aws ec2 describe-instances

Setting Up VNC Server on RHEL

This post describes the steps necessary to set up a Red Hat Enterprise Linux (RHEL) server on Amazon Web Services.

Launch EC2 Instance

Nothing special. Just launch a free tier eligible t2.micro with all the default. You do not need to modify the security groups because you will VNC over port 22.

You can do the following after launch, but it is easier to do in the user data section of the ec2 launch. Thus, add this to the user data:

#!/bin/bash
# Install GUI and VNC packages
yum update -y
yum -y groupinstall "Server with GUI" Desktop 
yum -y install tigervnc-server xterm

# Set default run mode to GUI
systemctl set-default graphical.target

Save Your Details

Save your private key to key.pem. Don't forget to chmod 400 key.pem.

Note your server's public DNS address as servername.

Set Up the VNC Server

SSH to your server as ec2-user.

Wait until all the installs are complete. Then restart the server

sudo tail -f /var/log/messages
sudo shutdown -r now

Once it has rebooted, log back in. Create a password for ec2-user. You may need this if the screen saver locks.

sudo su -
passwd ec2-user

Then, create a password for the VNC connections. Only the first 8 characters matter, and it will not be stored securely on the server.

vncpasswd

Put the following in the ec2-user's ~/.vnc/xstartup file:

#!/bin/sh
( while true ; do xterm ; done ) &
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
exec /etc/X11/xinit/xinitrc
[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
twm &

Connect to Your Server

From your client, establish an SSH tunnel to your server over which VNC will be securely routed. After you execute this command, any connections from your client machine to your client machine's port 5900 will be forwarded over an SSH tunnel and delivered to your server's port 5901.

ssh -i key.pem -l ec2-user -C -L 5900:localhost:5901 <servername>

Start a VNC server with:

vncserver

Troubleshooting

Logs

/var/log/messages
/home/ec2-user/.vnc/ip-172-31-92-87.ec2.internal:N.log

SSH

ssh -v gives you debug info on the ssh connection.

The .vnc log file will tell you what port the connection is open on. That should be the second port number in your ssh tunnel command.

To verify the GUI install:

From the AWS console EC2 panel.
Select the EC2 instance.
Actions->Instance Settings->Get Instance Screenshoot

You should see a mouse pointer, not a login prompt.

Other

You may need to remove lock files from tmp

sudo rm /tmp/.X11-unix/X*

Remove log and pid files:

rm ~/.vnc/*.log ~/.vnc/*.pid

Reboot

sudo shutdown -r now

References

RHEL Installation Guide

Fantasy English Premier League Overview

Football vs Soccer

In college, I mentioned spending a lot of time planning how I would escape to Mexico if I had to go on the lam some reason, like if I was framed for a murder or if I had to avenge some wrong by committing a crime myself. My argument-prone Canadian roommate said, “Or Canada. You could escape to Canada.” Another roommate and I then spent the next half hour arguing with him that it wasn’t a competition and that clearly Mexico was the better destination if one was an international fugitive.

Well, in the case of football vs soccer, I will finally throw a Canadian a bone and honor his beloved sport by calling it football. I give negative shits about the NFL, so I have no problem calling his sport by the name the rest of the planet uses.

The English Premier League (EPL)

The EPL is twenty teams at the top of the English Football League (EFL). The English Football League is a hierarchy of leagues consisting of teams from local amateur clubs all the way up to sponsored, professional teams. After each season, teams move up and down in the leagues based on performance. The EPL drops (relegates) the lowest three teams and brings up (promotes) three new teams each year. Lower level teams are not feeder teams for higher levels like in baseball minor leagues, but players do move up and down. The EPL is also called the premiership, the premier league, or Barclays premier league (BPL).

Current Teams

There are 20 teams in the EPL. The teams for this season include 19 from England and 1 from Wales and are ranked as follows:

  1. Manchester United
  2. Liverpool
  3. Huddersfield Town
  4. Manchester City
  5. West Bromwich Albion
  6. Chelsea
  7. Watford
  8. Southampton
  9. Tottenham Hotspur
  10. Burnley
  11. Stoke City
  12. Everton
  13. Swansea City
  14. Newcastle United
  15. Leicester City
  16. Arsenal
  17. Brighton and Hove Albion
  18. Bournemouth
  19. Crystal Palace
  20. West Ham United

Season and Games

This season goes from August 2017 to May 2018. During the season, every team plays every other team twice (once home and once away). Teams get 3 points for games won and 1 point for ties. Total points determines seeding for the tournament at the end of the season. That tournament, called the EFL Cup includes the EPL and the next 3 leagues down in the EFL. The winner gets money, ass, and included in European leagues and/or tournaments.

Note: British folks use the word fixture for what we would call a game, that for a specific match between two teams at a specific time and place.

Watching Games

Games are usually played on Saturday and Sunday. I couldn’t figure out what times they are usually on. The UK is 5 hours ahead, 6 after the Fall time change.

TV: In the US, NBC has exclusive rights to air EPL fixtures. The games show up on various NBC-owned stations, including things like the USA channel. World Soccer Talk keeps an up to date list of which games are being shown on which stations each week.

Internet: It looks like the cheapest way to get access to a lot of games is the Premier League Pass from NBC Sports Gold. It’s $49.99 for the season. You get access to 130 streamed games (out of 380) plus highlights and some other nonsense. See the press release. There are probably lots of other shady ways to stream them also, but I don’t know how. I’m not above shadiness, though, so feel free to pass on any links you have.

Fantasy League Teams

We are play on Togga, which probably means something to British people. There, you have a team of 16 players. Each week, you earn points based on the performance of the 11 players you choose to start that week. The players you can start is limited by their fielding positions. You can’t start 11 strikers, for example. The list of allowable forward-mid-defender formations are:
3-5-2
3-4-3
4-5-1
4-4-2
4-3-3
5-4-1
5-3-2
You also have to start a goal keeper every game. I don’t know if you can change formations each week or if you pick a formation per season. Probably you can change week to week.

Auto-subs: If one of your designated starters plays 0 minutes, then the system automatically subs in the highest scoring player on your bench in the same position as the 0 minute starter. This is an optional feature that has to be enabled by the league Commissioner. We have a bunch of newbs like me playing to we allow auto-subs. It’s likely we will each screw up at least once and start an injured player or something ridiculous.

Fantasy League Points

On Togga, players earn points each week based on their performance in 18 different categories, like scoring a goal, interceptions, clean sheets. Clean sheets, btw, are games in which the player plays 60+ minutes with no goals scored during those minutes. While each player can theoretically participate in all 18 events, it is important to consider the likeliehood of the event per position. For example, yes a goalkeeper gets 10 points if he scores a goal, but it’s far more likely that he will accrue 8 points for a clean sheet.

Goal

The goal is to accrue the most points in a season compare to the rest of the people in our league. It starts with acquiring players in a draft. Then nominating players for games and trading players throughout the season. I have an additional goal of using this to learn TensorFlow, but I will discuss that in a later post. In my next post, I will cover the draft.

Where Everybody Knows Your Name

Thinking about the erosion of anonymity. Facial recognition will identify us as we walk down the street. Our phones already track us. China is closing in on phasing out cash. It is getting increasingly hard to remain anonymous in the physical. In the digital world, forget about it.

But is that so bad? We once lived in small town where everyone knew who you were, and you knew who everyone was. It is only with the advent of modern, large cities with reliable transportation systems to take us far from home each day that we found some anonymity. Thus the erosion of anonymity is just a return to the good ol' days when everyone knew who you were, right?

Wrong. The difference is that in the past, you also knew everyone. You knew if you could trust them. In this era, corporations know who you are, but you know nothing about them. You don't know what designs they have for your or your wallet, but they know everything about you and what you do with that wallet.

Do we want to live in a society when every bus stop advertisement knows every sexual fetish (SFW) you've ever considered?

I have no solution, only the observation that the loss of anonymity is only a return to simpler times when the loss is symmetric.

Cheers,
Robert

VMWare Networking

vSwitches

vSwitches are network switches implemented in software. They interact with both virtual and physical network interface cards (NICs). In addition to the functions of a physical switch, they offer some unique functions like port groups.

VMWare offers two types of virtual switches on ESX servers: standard and distributed.

Standard switches

  • Operate on only one ESXi host. 

Distributed switches

  • Span multiple ESXi hosts.
  • Appear to VMs as a standard switch.
  • Enables VM migration between ESXi hosts without reconfiguration.

Uplink Adapters

Uplink Adapters are ESXi host's physical NICs connected to the outside network. vSwitches use uplink adapters to route traffic on/off the ESXi's local network.

yum

EPEL

Extra packages for enterprise linux (EPEL) is a Fedora effort. It collects packages that are not part of the core Red Hat distribution, but are useful. The packages for EPEL 5 and 6 are different because the core distributions changed.

For example, RHEL 5 did not include git but RHEL 6 does. Consequently, EPEL 5 contains git, but EPEL 6 does not. Interestingly, the git version in EPEL 5 (1.7.4) is greater than the git version in base 6 (1.7.1).

Repositories

yum repositories store and provide RPMs. They can be hosted locally or remotely. 

Client Configuration

Yum looks for repository configurations in:

  • /etc/yum.conf
  • /etc/yum.repos.d/*.repo

Yum looks in those files for a [repository] section, where repository is a unique identifier. Under that header, put the following directives:

  • name=repository_name
  • baseurl=repository_url
    • repository_url=[http|ftp|file]://[/][username@]fqdn/path/to/repo
    • usually baseurl=http://path/to/repo/releases/$releasever/server/$basearch/os/
      • $releaseserver=distroverpkg from /etc/yum.conf
      • $arch=<result from Python's os.uname(). note, in python, you have to import os first.>. usually either i686 or x86_64.
      • $basearch=$arch, usually.
  • enabled=[0|1]
    • 0 ignore
    • 1 include

For more.

Server Configuration

This creates a local repository. I'm not sure yet how to set up an http one. I think it involves installing httpd, which I am loathe to do.

  1. Install createrepo package: yum install createrepo
  2. Create the directory to hold the repo: mkdir -p /opt/yum/repo/<reponame>
  3. Copy the files for the repo to the directory.
  4. Tell yum to set up the metadata: createrepo /opt/yum/repo/<reponame>
  5. Configure the client with baseurl=file:///opt/yum/repo/<reponame>

VLAN

Networking Refresher

To understand VLANs, you have to remember some stuff from networking 101

Note: this is all Ethernet, IP specific.

Open Systems Interconnection (OSI) Layers

Physical Layer [OSI Layer 1] - Concerns physical transmission, including collision detection, voltage, etc.

Data Link Layer [OSI Layer 2] - Concerns transmission between two nodes in the same broadcast domain. Consists of two sublayers, media access control (MAC) and logical link layer (LLC).

Ethernet [OSI Layer 1 and 2] - A "family of technologies" for creating local area networks (LANs) defined by the 802.3. An Ethernet packet is

  1. Layer 1 synchronization bits
  2. Ethernet Frame
    1. To/from MAC addresses
    2. VLAN ID (optional. from 802.1Q)
    3. Payload (contains upper layer packets)
    4. Other, misc.

Network Layer [OSI Layer 3]- IP layer.

Networking Devices

Port- The little hole into which you plug a cable. Duh.

Hub, Repeater [OSI Layer 1] - A device that basically physically splices wires together. May be powered to amplify signals.

Switch [OSI Layer 2] - Like a hub except ports aren't connected by default. A switch know which ports reach which MACs. Switch only forwards packets to ports based on the to: MAC address. Exceptions (see below) are broadcast transmissions and VLANs.

NOTE: Unfortunately, the term switch has become overloaded to include much more sophisticated devices.

Router [OSI layer 3] - Like a switch except it also knows which ports map to IPs (or the next hop).

Network Domains

Collision domain [OSI Layer 1] - The set of devices whose network transmissions can collide on a physical medium, including through hubs. The network's physical configuration (wires, hubs, repeaters, etc.) defines the collision domain.

Switches treat each port as its own collision domain. E.g., A device on port 1 sends a message bound for a MAC on port 2. Simultaneously, a third device on port 2 transmits a message. In a two port hub, the transmission would collide. On a switch, the switch buffers the message from port 1 until port 2 is clear.

Broadcast address [OSI layer 3] - The highest IP address in a subnet. E.g., in 192.0.1.0/24, it's 192.0.1.255. Because the OSI layers are separate, the broadcast address says nothing about destination MAC addresses.

Broadcast domain [OSI layer 2] - The set of devices reachable by a broadcast transmission, including the sender. All devices connected to a switch are part of a broadcast domain, with the exception of tagged VLAN packets (see below). Since switches route to MACs, not IPs, and broadcast addresses are to IPs, not MACs, switches must send broadcast messages to all ports. The downside: two subnets on the same switch but different ports receive each other's broadcast messages.

When routers were expensive and a local area network (LAN) might have a lot of hubs but only one switch, all devices on the LAN were part of the same broadcast domain.

Routers define broadcast domain boundaries. Since routers are IP address aware, routers only forward broadcast messages to ports with IPs in the broadcast domain.

Broadcast traffic- Traffic addressed a broadcast domain's broadcast address.

Examples:

  • Services announcing their presence, e.g. file and print servers.
  • Routers announcing the routes they serve.
  • New-born machines seeking BOOTP or DHCP.

Issues:

  • Wasteful/congestion.
  • Misuse or misconfiguration leading to lots of congestion.
  • Insecure. Facilitates passive network mapping.

Virtual LANs (VLANs)

Now, let's talk about VLANs.

VLANs [OSI layer 2] - VLANs are a logical segmentation of traffic enforced by switches.

A Ethernet-frame header-field called a VLAN ID tags traffic as part of a specific VLAN. Both switches and hosts can apply/remove tags.

Each switch port can be tagged for the use of one or more VLAN IDs. Switch ports are usually tagged for a default VLAN ID. Network admins must make changes as necessary.

A trunk is a direct connection between two switches. Trunks can be chained across multiple switches. Hosts can operate on the same VLAN connected to different switches as long as the trunk (or chain of trunks) are tagged for the VLAN.

As mentioned, routers don't forward IP-level broadcasts between ports but switches do. With VLANs, switches limit VLAN-tagged broadcasts to ports tagged for that VLAN. This is a useful trick because switches are cheaper and faster than routers.

Reasons to use:

  1. Control services - Control which hosts use which broadcast-based services on a switch, e.g., limiting DHCP to certain VLANs.
  2. Save money - If you are trying to reduce the impact of broadcast traffic, VLANs let you use a switch instead of a router.
  3. Security 
  • Restricts breadth of broadcast information dissemination.
  • Preserves subnet integrity. Even if a user changes his machine's subnet, if he is on the wrong port, he won't be able to join the subnet. 
  • Mitigate broadcast storm impact.

Questions

None-the-less, I still have a number of questions:

Broadcast traffic

  1. Where is all this terrible broadcast traffic coming from in the first place?
  2. Why not cut it off at the source by muting/curtailing chatty broadcasts?
  3. It seems lots of Windows services broadcast. Is that the problem?
  4. How much of a problem is ARP?

tmux

Terminal Multiplexer

Like screen but better. Except I never used screen, so, I don't know what that means.

You may have to yum install it.

Pretty Pane Dividers- Using PuTTY, set the Windows->Translation->"Received data assumed to be in which character set" drop down box to UTF-8. Otherwise, the pane dividers just ain't gonna look right.

Cheat Sheet

Everyone's got one. Here's mine. All of these assume the prefix. Usually CTRL+b

%
Vertical pane split
"
Horizontal pane split
x
Kill pane
c
New Window
&
Kill Window
setw
alias for set-window-option
=
display paste buffers

Config File

See it on github.