The shell is your friend. But many developers don’t really know the shell, the Unix or Linux command-line environment available in several operating systems. (Bash is the best known, but there are others.)
Some of you, when you transitioned from Windows to Mac, took your (slow) clickety habits with you, not realizing that the power laid in that app called Terminal hidden under Applications somewhere. Some of you have been shelling into “the server” to tweak a setting or two without realizing that you could automate your life away without even cracking a devops tool.
Whatever brought you to the shell, chances are you’re not using it to its full advantage. Here are my top nine tricks for doing so:
1. Kill it with grep
The killall command never seems to kill what you want it to kill, so use grep instead. First, start with ps -awxu | grep -i civ | grep –v grep. That confirms that the pattern sent to grep is real and doesn’t kill the wrong thing. Now add | cut -c 9-14 to the end. This gives you a list of process IDs (pids), which are the handles your operating system uses to identify its running tasks (aka processes).
So now make the full command: kill -9 `ps -awxu | grep -i civ | grep -v grep | cut -c 9-14`. That kills all the instances of the latest incarnation of Sid Meyer’s little time suck. (On a Mac use ps -ef and cut -c 8-12 or so. Happy killing things … selectively!
2. Use Findinjar to discover which !@#$ jar file that class file is in
I’m sure this exists in other languages, but after a few weird NoSuchMethod or other types of exceptions in Java, you get to go hunt which .jar files have which class files. Or sometimes you just want to package some dependencies without all the mess. That’s where Findinjar comes in. There are many versions of this little shell script, and the one below is as good as any. You can also make versions for Zip or other types of archives.
#!/bin/bash
# first argument: file
# second argument: searchstring
#
find "$1" -name "*.jar" -exec sh -c 'jar -tf {}|grep -H --label {} '$2'' \;3. Use find … grep to, well, find things
Finding things is important. Sometimes, you’re looking for some file pattern like *.txt, and sometimes you’re looking for which files contain some string. These come up a lot. That’s where find … grep comes into play:
find . -name "*.txt" -exec grep -Hi somestring '{}' ';'
Replace the . after find with a directory name if you want to start somewhere besides the current directory.
4. Use Telnet as your browser
“What the frak,” you ask? Telnet? No, you shouldn’t use Telnet to shell into servers like in the old days. However, sometimes Curl and/or your web browser aren’t acting right when connecting to some endpoint. So try Telnet instead. For example, Telnet to Google.com on port 80, type GET / HTTP/1.1, and press Enter twice. With that, I will see for sure that Google.com is still up and listening on port 80. Is Oracle’s tnslistener really listening on 1521? Now you know what to do.
Here’s an example Telnet browsing session; the part you type is bold italic.
andrewoliver$ telnet google.com 80
Trying 216.58.217.110...
Connected to google.com.
Escape character is '^]'.
GET / HTTP/1.1
HTTP/1.1 200 OK
Date: Tue, 13 Jun 2017 15:08:36 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: NID=105=eqK2AqJE_AMNUX34Nu9XHjkU5H9yvb8ZuuEx4f8rCwtTZVmWkkO_FU_L7WrlAvDP8kiNzFS2Dyvrskd8jDghSxgGwudjA16Ubw05Gg3M1TyMEC7cLzMjLN0ovKp4mLuDOVBnOE2NIPbpkpGZ; expires=Wed, 13-Dec-2017 15:08:36 GMT; path=/; domain=.google.com; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding
Transfer-Encoding: chunked
8000
<!doctype html>
{snip}
5. List it backward with tac
Sometimes, you want to list a file backwards. cat lists it forwards. The tac command (cat backwards) lists it backwards. It is built-in on Linux, but If you’re on Mac you have to install Homebrew Coreutils. Honestly, after years of working on Linux, I’d never heard of this despite needing it on more than one occasion until a “stump the chump” session at Red Hat Summit revealed it to me.
6. Use GDrive to work with Google Drive
I actually use GDrive, which lets you pull from Google Drive, so some nontechnical users can maintain a list of emails and passwords that are automatically translated into a htpasswd file for Apache HTTP Server. Those users just edit a text file. And we don’t even talk anymore.
Here’s an example from a script using GDrive:
#check to see if the file has been modified (I store this in an env variable) gdrive info "$pwfileid" | grep Modified #Get a list of files ordered by their last modified time gdrive list --query "'$gdfileid' in parents" --order "modifiedTime" #grab a password file gdrive download --stdout "$pwfileid" #grab an entire directory gdrive download -r --path "$GOOGLEDIR" "$gdfileid"
7. Automate Git via shell
I use Git to store many things. Some of those things are scripts or things that need to be bundled into a download if they change. Although Jenkins is awesome, I don’t need a whole build server to see if I need to pull down a new copy of a config file, shell script, or Zip file for download.
Here’s a little bit of shell trickery I pulled originally from Stack Overflow.
git remote update
# magic to check if our copy is the same as the upstream copy
UPSTREAM=${1:-'@{u}'}
LOCAL=$(git rev-parse @)
REMOTE=$(git rev-parse "$UPSTREAM")
BASE=$(git merge-base @ "$UPSTREAM")
# if our github copy needs a pull then we'll redo the distro
if [ $LOCAL = $REMOTE ]; then
echo "Up-to-date"
elif [ $LOCAL = $BASE ]; then
echo "Need to pull"
elif [ $REMOTE = $BASE ]; then
echo "Need to push"
else
echo "Diverged"
fiIt lets you see if you need to pull or push, or if you’re just totally out of whack (stop editing on the server!). You can fill in some actions where the echo commands are and make this do something more useful.
8. Separate values the easy way
If you Google “parse a comma-delimited string,” you’ll find all kinds of things about changing the internal field separator. This works for some people, but I find it error-prone. That’s why I use the tr command. With it, you can loop through anything separated by anything. I’ve generated dozens of Pig scripts from shell scripts that read delimited files made by analysts in Excel using nothing more than a few lines of shell script.
#!/usr/bin/env bash
input="somestring|stringtwo|stringthree
yetanotherstring|stringfive|stringsix"
somestrings=$(echo $input | tr "|" "\n")
for astring in $somestrings
do
echo "[$astring]"
Done a
ndrewoliver$ ~/foo.sh
[somestring]
[stringtwo]
[stringthree]
[yetanotherstring]
[stringfive]
[stringsix]9. Make SSH tunnel your spirit animal
I recently installed something on EC2 that actually required X-Windows. I was pretty miserable about the whole thing. However, if you install XFCE (a minimal window environment) and VNC, you can tunnel everything over SSH so it is actually encrypted and minimally painful. I’ve used SSH tunnels for a lot of things. Basically, it makes a port on your laptop that gets sent over SSH to a port on the server. You then connect to the localhost port (in this case, 5901):
ssh -L 5901:localhost:5901 -i ~/mykeyfile.pem ubuntu@myec2hostname
Ask any old Unix hacker, and he’ll have his own special tool set just as I have these nine. For whatever reason, these are the things I use most. As for me, I may or may not have generated this whole article via a shell script.