Recipes for tm-diff (Time Machine difference)

A few recipes for using the tm-diff.sh script that I posted earlier (see link for details):

• Ignore errors/warnings (ex: "Permission denied"):

  tm-diff.sh folder1 folder2 2> /dev/null

  
  
• Find all files, regardless of their privileges/permissions:

  sudo tm-diff.sh folder1 folder2

  
  
• Display disk usage, in "human-readable" sizes (ex: KB, MB, GB...):

  tm-diff.sh folder1 folder2 | while read f; do du -h "$f"; done

  
  
• Display disk usage, with a grand total at the end (sizes in KB):

  tm-diff.sh folder1 folder2 | while read f; do du -k "$f"; done | awk '{ sum += $1; print; } END { print sum; }'

  
  
• Display the largest 50 files (sizes in MB):

  tm-diff.sh folder1 folder2 | while read f; do du -m "$f"; done | sort -n | head -50

  
  
• Display the smallest 100 files (sizes in KB):

  tm-diff.sh folder1 folder2 | while read f; do du -k "$f"; done | sort -rn | head -100

  
  
• Display a full listing (with mode/privileges, owner, group...):

  tm-diff.sh folder1 folder2 | while read f; do ls -l -h "$f"; done

  
  

Notes:

1. This presumes you've set your PATH so it'll find the tm-diff.sh script. (Or use its full path.)


2. In the commands above, "folder1" and "folder2" of course represent the paths to the two Time Machine backup folders to compare; they'll look something like:

   /VOLUMENAME/Backups.backupdb/USERNAME/2009-07-20-091153



3. The script explicitly searches for regular files only; no directories or other special files.


4. I use a read loop, since this will get the full path, including "special" characters like spaces and double-quotes.


5. Sizes are in the classic style ("as God intended"): KB = 1024 bytes; MB = 1024 KB; GB = 1024 MB ...


6. Most Time Machine backups contain tens of thousands of files, so these will take awhile. The ones that sort will display no files, until all of them have been found, to be sorted - so they will seem to take even longer.

Show differences between two Time Machine backups

Below is a quick Bash script to display the difference between two Time Machine backups.

The output is simply the pathnames of the files that were added to Time Machine in that time period. This may of course be piped to another command/script to provide further info - say, the amount of space used.

Cheat: Provide a simple date (formatted the same way that Time Machine does) instead of the "older" backup pathname, and it'll work whether or not there was a backup then. (The order of the arguments is unimportant; the script will figure it out.)

#!/bin/bash

LF=$'\n'
usage="Usage: $(basename $0) Time-Machine-folder-1 Time-Machine-folder-2"
usage=$usage$LF"(Compare the two Time Machine backup folders;"
usage=$usage"find and display files that were backed up in that time period.)"

function tweakDate { # to a fmt understood by find cmd
   echo "$1" | awk -v FS="" '{print $1$2$3$4$5$6$7$8$9$10" "$12$13":"$14$15":"$16$17}'
}

if [ ! $# == 2 ] ; then
{
   echo "$0 requires two arguments; the Time Machine folders to compare." >&2
   echo "$usage" >&2
   exit 1
}
fi

baseName1=$(basename $1)
baseName2=$(basename $2)

# NB: operator must be escaped or will signify redirection
if [ "$baseName1" \< "$baseName2" ] ; then
{
   olderFldr=$1
   recentFldr=$2
   olderFldrName=$baseName1
   recentFldrName=$baseName2
}
else
{
   reverse=TRUE
   olderFldr=$2
   recentFldr=$1
   olderFldrName=$baseName2
   recentFldrName=$baseName1
}
fi

if [ ! -d "$olderFldr" ] ; then
{
   echo -n "Warning: $olderFldr doesn't seem to exist;" >&2
   echo " maybe you're just using it to specify a date?..." >&2
}
fi

olderFldrParent=$(dirname $olderFldr)
recentFldrParent=$(dirname $recentFldr)

if [ ! $olderFldrParent == $recentFldrParent ] ; then
{
   echo -n "Warning: These folders are not in the same parent folder;" >&2
   echo " this may not be what you want..." >&2
}
fi

olderFldrDate=$(tweakDate $olderFldrName)
recentFldrDate=$(tweakDate $recentFldrName)

if [ ! -d "$recentFldr" ] ; then
{
   latestBackup="${recentFldrParent}/$(ls -1 $recentFldrParent | tail -2 | head -1)"

   echo "Can't find $recentFldr; searching the latest backup" >&2
   echo "    ($latestBackup)" >&2
   echo "    for files modified between $olderFldrDate and $recentFldrDate." >&2
   echo "    This may take awhile..." >&2
   echo $LF >&2

   find -x "$latestBackup" -type f -newermt "$olderFldrDate" \! -newermt "$recentFldrDate" -print
}
else
{
   echo "Now searching for backups in $recentFldrName" >&2
   echo "    that are more recent than $olderFldrDate." >&2
   echo "    This may take awhile..." >&2
   echo $LF >&2

   find -x "$recentFldr" -type f -newermt "$olderFldrDate" -print
}
fi

# eof

Cloudera Desktop setup tip: DNS, DNS, DNS

Yes, it's important with Cloudera Desktop, just like everything else - try to fudge domain names and get bit.

Strange errors like:

"An unknown error occurred: hdfs put returned bad code: 255 stderr: 10/02/18 18:41:52 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020... Bad connection to FS. command aborted."

When attempting to upload a file - and I know that localhost is indeed responding properly on that port because it works find from the command line.

Solution: Spend the few minutes to determine what the real domain names are (*1) and set up the config files (*2) properly.

*1: ifconfig will tell you the IP address of the node you're on; host will tell you the domain name. If there is none, see *3 below.

*2: config files of potential interest:

/usr/share/cloudera-desktop/conf/cloudera-desktop.ini

/etc/hadoop/conf/masters

/etc/hadoop/conf/slaves

/etc/hadoop/conf/core-site.xml

/etc/hadoop/conf/hdfs-site.xml

/etc/hadoop/conf/mapred-site.xml

/etc/hadoop/conf/hadoop-metrics.properties

/etc/hadoop/conf/hadoop-env.sh

/etc/hadoop/conf/configuration.xsl

/etc/hadoop/conf/fair-scheduler.xml

*3: No domain name? See whoever's in charge of DNS, to fix it. That you? Well, you can either do it right (a little effort up front, pays big in the long run...) or you can try to handle it via /etc/hosts - good luck with that though.

Note to self: Don't shortcut DNS!

Snow Leopard pestering for keychain password

Interesting new behavior in Mac OS X 10.6 (Snow Leopard):

Services that formerly required a single authentication of the keychain (at launch) now ask every time, if the keychain is locked.

I've noticed this at least with Apple Mail and Google Contact Sync (ex: the gconsync process asks access the keychain).

The new behavior is certainly more secure (formerly, the passwords read from the keychain had to be stored somehow, to use later, thus providing an additional potential place to steal them). However, I don't agree it's worth the trade for the level of bother.

And it forces me to either:

1) Type my keychain password frequently (some risk there; ex: if someone's watching).

OR

2) Adjust the automatic keychain locking to be less frequent - which could well result in it never being locked (since it would continually be a accessed, resetting the countdown to automatic locking).

I didn't like either, so I found a way to tell Mail to check (poll) less frequently; every six hours:

defaults write /Users/mvgfr/Library/Preferences/com.apple.mail PollTime 360

(The above is to be issued to a command line prompt, all on one line. If you're not familiar with the command line, here's a beginner's guide. Standard warnings for the command line apply; if you're not careful you can do serious damage.)

This works for me, since I'm using Mail only as a backup of my mail messages; I compose and read mail in other ways.

(The Mail Preferences window allows a maximum of 60 minutes - and this is what shows when set as above, though the custom setting is thankfully maintained and not overwritten.)

Google Contact Sync (gconsync) took a little more effort; documented at the previous link. The concept may apply to other types of synching, though would require changing another parameter, since the above is specific to Google Contact Sync.

Decrease Address Book sync frequency (And some keychain tips)

Snow Leopard's "new" Address Book sync with Gmail's Contacts works great - though I'm not sure why it defaults to sync every hour; I certainly don't change contact info that frequently. :)

If you keep your keychain locked most of the time* being bothered to authenticate OR click cancel, twice, is a needless bother.

So, here's the command to set the frequency to whatever duration you like (in seconds); I chose a full day, since that's more than enough for me:

defaults write ~/Library/LaunchAgents/com.google.GoogleContactSyncAgent StartInterval -int 86400

It's also apparently necessary to log out and back in, to get the change to take effect.

*Keychains can hold some pretty sensitive info. If you don't have the keychain lock itself after X minutes inactivity, you should consider it. Pop open the "Keychain Access" app (in /Applications/Utilities/) and take a look at what's in your own keychain - sure you want to leave that unlocked? If not:

• Click the "Edit" menu
• Select "Change settings for Keychain 'login'..."

  I recommend turning both checkboxes ON and setting a short lock time.

Now, once you've done this, you may find that Safari is pestering you mercilessly, to authenticate. This is because it's being very careful and storing virtually everything you type in Safari, in the keychain - since it can't tell how sensitive everything is, though it knows that some of it might be credit card numbers and the like. Fortunately there's a solution that's both secure and convenient; tell it not to store that info:

• Click the "Safari" menu
• Select "Preferences..."
• Select the "Autofill" tab
• Turn OFF the checkbox for "Other forms".

  (If you're curious, you can see what it's been storing by clicking the "Edit..." button to the right of the checkbox. It's mostly web searches and other such innocuous stuff. As you can see by the other checkboxes, passwords are stored separately.)

"So don't do that"

Interesting... :)

I just upgraded to Mac OS X Snow Leopard and coincidentally started trying to dig into why my MacBook has been running out of battery "early", shutting down ungracefully.

Of course it just starts right back up and the worst I'm left with, is trying to figure out where I was. Which I make unnecessarily worse by stopping to check to see if there is anything in the logs, and getting further distracted. There never is - except today. Excellent. :)

I've been following Apple's docs on this and running the battery right down to the end, to calibrate it. This morning I made it happen again and when I jacked back into AC power, it started up just fine - and entered an infinite loop.

Strange; it looked like it was just about to display the Login Window, but then went back to the old-school text screen that comes before that - and then started cycling back and forth. Drat.

Double drat: I'd turned off ssh ("Remote Access" in the Sharing Preferences Pane) and so, couldn't ssh in, to check what was going on.

So I powered off and tried a shift-boot ("Safe Mode"), but no go; same behavior.

OK; So I booted into Single User Mode and started spelunking...

Several tangets later :) I saw a log message; something that was consulting /etc/authentication* had failed with an "unexpected character"... Sure enough, that file was hosed. (More on that in a moment.)

So... I pulled that file off the Install DVD and was on my way. (Running the Installer from DVD - amazingly well-written! - may have worked, though I didn't try.)

*More on /etc/authentication:

• Interestingly, /etc/authentication is a dynamic file; it was modified on reboot. The changes were not huge, though interesting to track down manually via the XML, since the order changes (which is not important to XML).
  
• This file is critical in determining who (or what process) is allowed to do what - and since it contained invalid data, Mac OS X could not even get to the Login Window.
  
• My fault: I had inadvertently caused etc/authentication to be at risk by running system_profiler in a tight shell loop, to eat battery (since it not only spews data, but also consults hardware) and it (I have since learned :) consults /etc/authorization - which my command had kept open (and vulnerable) FAR more than normal. Live and learn. :)

Snow Leopard "safe sleep" tweak

A quick bash script to dynamically adjust the sleep behavior under Snow Leopard - to avoid the expense of maintaining the sleepimage if the battery is above a certain level.

Thanks again to Joe Kissell for the original script from (TidBITS#893/20-Aug-07).

#!/bin/sh

#
# 20090902 mvgfr: rules have changed (ex: hibernatemode; there is no 7) so recode
#                 AND be more fault-tolerant...
#                 (setpoints raised, & cron freq increased, since lately been losing power before sleep)
# 20090901 mvgfr: Snow Leopard now defaults to "Secure Virtual Memory" so change safe mode to 7
#    (need programmatic way to check, anything better than system_profiler?)
# 20090728 mvgfr: reduce setpoints by 10 (%)
# 20070908 mvgfr: impl setpoints, tweak msgs, Trash vs rm, debug...
# original: Joe Kissell ,  (TidBITS#893/20-Aug-07)
#

debugMe="" # simple toggle; empty means NO & anything else means YES

setpointLow=20
setpointHigh=25

MODE=`/usr/bin/pmset -g | awk '/hibernatemode/ { print $2 }'`
LEFT=`/usr/bin/pmset -g batt | grep Internal | awk '{ print $2 }' | awk -F % '{ print $1 }'`

if [ $LEFT == "(removed)" ] ; then LEFT=0; fi # catch case in which no batt is seen

if [ $debugMe ] ; then /usr/bin/logger -t "hibernatemode" "current mode (on entry): $MODE; batt % remaining: $LEFT"; fi

if [ $LEFT -le $setpointLow ] && [ $MODE == 0 ] ; then
  {
  /usr/bin/logger -t "hibernatemode" "Battery level ${LEFT}%; setting hibernatemode ON"
  /usr/bin/pmset -a hibernatemode 3
  }
elif  [ $LEFT -ge $setpointHigh ] && [ $MODE != 0 ]; then
  {
  /usr/bin/logger -t "hibernatemode" "Battery level is ${LEFT}%; setting hibernatemode OFF"
  /usr/bin/pmset -a hibernatemode 0
  mv /var/vm/sleepimage /Users/mvgfr/.Trash
  }
fi

if [ $debugMe ] ; then
    MODE=`/usr/bin/pmset -g | awk '/hibernatemode/ { print $2 }'`;
    LEFT=`/usr/bin/pmset -g batt | grep Internal | awk '{ print $2 }' | awk -F % '{ print $1 }'`;
    /usr/bin/logger -t "hibernatemode" "current mode (on exit): $MODE; batt % remaining: $LEFT";
fi

#end

macports notification of outdated

A quick and dirty bash/shell script to get automatic notification of outdated macports:

#!/bin/bash

# 20090902 mvgfr: tweaks
# 20071110 mvgfr: check for outdated macports & notify if any - via growl & stdout (emailed when cron'd)
# 20080209 mvgfr: must run sync before, so it can learn about updates!
#                 also added "-H localhost" workaround for growl Leopard bug (ignores some)

debugMe="" # presume we'll only use alphanum; like "YES" or empty string for NO

# may take awhile; is this an issue?
if [ ! $debugMe ] ; then /opt/local/bin/port sync; fi

if [ $debugMe ] ; then
    potentialOutput="`echo 'debugging:';ls -1|head -1`"
else
    potentialOutput=`/opt/local/bin/port outdated`
fi

if [ "$potentialOutput" != "No installed ports are outdated." ] ; then
    echo "$potentialOutput" # for cron to email to root
    if [ -f /usr/local/bin/growlnotify ]; then
 NumLinesToDisplay=$((`echo "$potentialOutput" | wc -l` -1))
 if [ $NumLinesToDisplay -lt 1 ]; then
     echo "underflow error; see log" | /usr/local/bin/growlnotify -H localhost -s -t 'macports outdated:'
 elif [ $NumLinesToDisplay -gt 20 ]; then
     echo "overflow error; see log" | /usr/local/bin/growlnotify -H localhost -s -t 'macports outdated:'
 else
  echo "$potentialOutput" | tail -`echo $NumLinesToDisplay` | \
      /usr/local/bin/growlnotify -H localhost -s -t 'macports outdated:'
 fi
    fi
fi

if [ $debugMe ] ; then echo "`date "+%Y%m%d-%H%M%S"`: $potentialOutput" >> /var/log/port-outdated.log; fi

#end

(Old 2007/12/01 script here.)

Snow Leopard upgrade notes

Words of warning for anyone who's customized "under the hood" a little:

1. The Snow Leopard installer (doing an upgrade) wipes out /var/log/.
   

   Not only would I have preferred to keep the old logs (historical reference; maybe to compare pre- and post- Snow Leopard), but I also had some custom stuff logging into there (ex: freshclam) that was just gone. And my code was quick-and-dirty so it didn't recover gracefully (ex: to the logs not existing) so I had to do a bit of cleanup.
   

   No huge deal (this is not-unexpected behavior for /var/) though it is new behavior for the Installer, so I'm adding this to the great Internet KnowledgeBase, so maybe it saves someone else a bit of effort. :)
   

2. It seems to have wiped out the /opt/local/mysql symlink to /opt/local/mysql-version_spec; also easily fixed.
   
   
3. MacPorts wouldn't selfupdate for me, so I just reinstalled from the latest disk image (1.8.0 for 10.6). I also installed the new Snow Leopard Xcode Tools (3.2; newer even than the version I downloaded last week!) on the Snow Leopard DVD.

BTW: If you have done some customizing, look for files with these strings added to the name (not just appended): "~previous", "~orig", or (maybe) "disabled". Such as:

• /etc/syslog.conf~previous
• /etc/postfix/main.cf~orig

As you may need to bring some customizations back. (Though in both the above cases, I was happy with the way the Installer had done it.)

(Thanks to Apple for putting some nice info in various logs.)

Otherwise, smooth sailing! Definitely a must-have upgrade, especially considering the price and what it sets up for the near future.

Macs are more secure

Check out the Macalope's column "Lies, damned lies, and statistics" in which he discusses yet another (strange) survey trumpeting trouble for Apple.

He also discusses Apple trumpeting about Macs being more secure, and makes a good analogy to be somewhat more realistic: "no matter how nice the neighborhood, you can only leave your houses unlocked for so long before something bad happens".

Clearly, no matter how secure Macs are to start, if you make insecure choices (ex: easy password, auto-login, etc.) you may find yourself in trouble.

The comments below the article are worth reading too; and indeed security does include many things.

One factor is the default config. In MS Windows, virtually everything defaults to "on" whereas the Mac defaults to "off"; you can't be attacked via a door that isn't there.

Another factor is monoculture vs. diversity; much malware depends on the targets running IE or Outlook or whatever - whereas the Mac market benefits from a different and more varied environment and so is more difficult to target.

Macs are certainly not immune (nothing is) - however they do suffer from far less malware and one reason is that they start out more secure than MS Windows. The rest is up to you.