shards mvgfr: geek: 2018

2018/12/10

How-To recover / export Outlook for Mac "Smart Folder" AKA "Saved Search" configs

Apparently there's no way to export this part of the Outlook config directly. :/

However there are some strings to pull on, which can help:

There are "links" to the config data, in a sqlite DB, which can be seen like so:

sqlite3 ~/'Library/Group Containers/UBF8T346G9.Office/Outlook/Outlook 15 Profiles/Main Profile/Data/Outlook.sqlite' 'select * from SavedSpotlightSearch'

(The "UBF8T346G9" above, may vary?)

Even better, the "Smart Folder" configs (including, apparently, those that have been deleted?) are stored in this directory (for recent versions of Outlook for Mac):
~/'Library/Group Containers/UBF8T346G9.Office/Outlook/Outlook 15 Profiles/Main Profile/Data/Saved Searches/'

As a cheap hack, here's what I did, to recover the searches from one Mac, and implement on another:

Dump the data; ex:

find ~/'Library/Group Containers/UBF8T346G9.Office/Outlook/Outlook 15 Profiles/Main Profile/Data/Saved Searches' -type f | while read -r thePath; do echo; echo; ls -ld "$thePath"; cat "$thePath" | tr -d '\000' | tr -cs "[:print:]" '\n' | egrep -v '^[[:blank:]]*$' | sed 's/\([^ (]\)(/\1\'$'\n(''/g' | sed 's/)\([^ )]\)/)\'$'\n''\1/g'; done

(Removes NULLs; translates non-printing chars to LFs; removes extraneous LFs; puts "sections" on separate lines - a bit hacky, since there's no docs that I can find, for this file format.)

Interpret the needed bits; more info below.

Some tips, on decoding the data:

The data is apparently in two-byte characters - the above is a cheap hack which works in ASCII anyway. :/
Near the top, is some definitory info, like the com_microsoft_outlook_folderID to search in. (You may be able to deduce the correct folders, without having to figure out how to determine which folder corresponds to which ID.)
Then the "Smart Folder" name.
Then, finally, the search / query itself - there's a bit "encoding" here too, so not quite "cut and paste". :/

And since (on a Mac) these are stored in the "Raw Query" format, which uses the underlying Spotlight metadata, here's the official Apple doc on the File Metadata Query Expression Syntax.

Interesting note: This exposes the "Raw Query" syntax (leveraging Spotlight / mdfind), which can be a learning opportunity, for creating more interesting searches (since Raw Query docs are a wee bit sparse).

2018/10/26

git-crypt works smoothly - until it doesn't

Some repos use git-crypt to encrypt secrets; they are committed in encrypted form, and decrypted locally, auto-magically, using GPG keys.

This magic is, shall we say, simple until it's not -- in "interesting" ways.

This is written from the point-of-view of someone inheriting an existing config - which broke, due to multiple keys.

In the hope of saving someone else their sanity, here are a few learnings.
(Which are hopefully even correct.)

We'll start with the easy stuff; then, well, Buckle up...

How to get started as a new collaborator:

brew install git-crypt
install the "GPG Suite"
generate a key pair
upload your new public key to the interwebs
give public key to an existing collaborator, who must:
- add the new user to their GPG keychain
- sign the new user's key
- git-crypt add-gpg-user --trusted USER_ID
  - Use '--trusted' to avoid dependency of public "Web of Trust"
  - The USER_ID above is usually the email address that the user configured their GPG keypair with
  - add-gpg-user should result in output like this:

[master 30babf07] Add 1 git-crypt collaborator
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 .git-crypt/keys/default/0/72E278AE2FB3...8F90BB21B36FD67.gpg

How was git-crypt set up in the first place?

(See above for a bit more detail on some of these steps, such as expected output.)

brew install git-crypt
navigate to the repo you want to use git-crypt with
git-crypt init
- Note: This creates a symmetric key.
add the first GPG user: git-crypt add-gpg-user --trusted ADMIN-USER_ID
- This user must exit already in GPG.
- You might consider this the "admin" user; they'll be the only one to be able to decrypt secrets, add more users, etc. - until other users are added.
unlock, using new GPG key (will prompt for that key's passphrase): git-crypt unlock
config a .gitattributes file with contents like secretfile* filter=git-crypt diff=git-crypt
- the .gitattributes file defines which files are to be encrypted
git add .gitattributes
commit and push: git commit -m 'your comment here'; git push

A few notes:

what causes the encryption to actually take place?
- See the notes on the .gitattributes file, above.
check encryption status (for encrypted files, GITCRYPT shows at the top):

git-crypt status -e | awk '{print $2}' | while read thePath; do echo $thePath\: $(cat $thePath | xxd -l 9); done
Warning: You could push the change and confirm it's encrypted in the web UI - except if it's not, that secret is now forever* ensconced in your repo. (*How to remove secrets from a repo - AKA: It's too late.)

GPG items in MacOS keychain, are not named with "GPG", but with "GnuPG"
we're using GPG here (not PGP); it makes little difference to the procedure (ex: a GPG fingerprint is not for GPG only)
if freshly cloned, need to git-crypt unlock again (default state is locked)
who's got access?

ls -l .git-crypt/keys
each filename contains a user's fingerprint (look that up, on a keyserver)

Trouble getting file to actually encrypt?
- git-crypt status -f
- git-crypt lock --force
- "Touching the file and re-committing it forces git-crypt to re-encrypt it"
Seeing errs like: "still unencrypted even after staging" OR "encrypted file has been tampered with" OR "Warning: one or more files is marked for encryption via .gitattributes but was staged and/or committed before the .gitattributes file was in effect" ?
- unstage (ex: git reset HEAD secrets.yml)
- redo the dance with git-crypt status -f and git-crypt lock --force
- maybe start from a fresh clone - and save aside, any files that are unencrypted
- be certain the file is really encrypted before using git add filename
But git-crypt status -e says the files are encrypted!

NO; it's only saying that those files are configured to be encrypted
check the contents to confirm if it's actually encrypted (see "check encryption status" above)

getting an err like ERROR! Unexpected Exception: 'utf8' codec can't decode byte 0xd0 in position 11: invalid continuation byte ?

Check your git-crypt config; if that's OK, reclone the repo (the git-crypt status may be hosed.)

How to reset the encryption on a repo:

Here be dragons; this should be avoided, but if you have to...

list files that have been encrypted: git-crypt status -e | awk '{print $2}' > encrypted-files
make sure repo is in UNlocked state: git-crypt unlock
save decrypted copies of all encrypted files; ex: git-crypt unlock; tar czf ../saved.tgz ./

if you don't have unencrypted copies anymore?
get them from another collaborator, old unlocked copy of the repo, ...
there is no known way to recover them otherwise

remove all encrypted files; ex: cat encrypted-files | xargs -t -n1 rm
remove all git-crypt files: rm -rf .git-crypt .git/git-crypt
? may be necessary to save & remove the .gitattributes file too? (doubtful)
commit: git commit -a -m 'your comment here'
re-config git-crypt: git-crypt init
- Note that this creates a new symmetric key, stranding files encrypted with any other key.
add the first AKA "admin" user: git-crypt add-gpg-user --trusted ADMIN-USER_ID
unlock, using new GPG key: git-crypt unlock
add any addtl users: git-crypt add-gpg-user --trusted ONCE-PER-ADDTL-USER_ID
if you removed the .gitattributes file above, copy it (or its contents) back
- the .gitattributes file must be in place BEFORE adding a file that must be encrypted.
copy decrypted files back in
confirm files are decrypted: git-crypt status -e | awk '{print $2}' | while read thePath; do echo $thePath\: $(cat $thePath | xxd -l 9); done
- It may be helpful to make the files different (ex: add a comment) to help force encryption with new key...
some extra git-crypt magic: git-crypt status -f
make sure repo is in UNlocked state: git-crypt unlock
force encryption: git-crypt lock --force
CONFIRM FILES ARE ENCRYPTED (they'll show GITCRYPT):

git-crypt status -e | awk '{print $2}' | while read thePath; do echo $thePath\: $(cat $thePath | xxd -l 9); done

if not, see notes above - it will be messy if you add (or worse commit) unencrypted info
for each of the encrypted files: git add ...
commit & push: git commit -a -m 'your comment here'; git push
you probably want to unlock again: git-crypt unlock
after keys are reset, a possible solution to: checkout (ex: of a branch) fails with "encrypted file has been tampered with":
- make a fresh clone of the repo
- leave it locked
- checkout branch (ex: keys were reset on master, but old keys are left on your branch)
- cherry-pick the commits for the new keys & newly-encrypted files (in chron order?)
- then unlock
a fresh clone is best; otherwise, something like this might help: git-crypt lock --force; git stash; git pull

WHY would you ever want to reset the encryption on a repo??

You somehow got secrets committed, with multiple symmetric keys (ex: ran git crypt init more than once).
You want to be safe, after a collaborator has left the project.

References:

2018/03/02

is your Mac waking on its own, draining battery and then just plain powering down?

I've twiddled every setting I can find, and this still happens:

I close the lid of my MacBook,
Carefully listen for the fan to stop...
(Wish they'd kept the pulsing LED; that was an excellent design.)
And only _then_ do I put it in my bag.
Some time later, I pull it out - to find it hot and/or powered off. Yuck. :(

(Kudos to Apple for constructing macOS so solidly, that even when it powers down unexpectedly (still sub-optimal), it actually does come back quickly - even with unsaved work, ready to pick up where you left off. Nice.)

Recently, I've been watching this strange sleep behavior even more closely, and several times have witnessed this sequence:

Close lid & listen for fan to stop...
Keep listening; after a few more moments, the fan starts again!
This sequence repeats

This seems to be new-ish behavior; as of the last few releases of macOS.

I've done everything I can find (details below), to get it to _stay_ asleep, but to no avail.

However I just found something new; from the man page of the pmset command:

ttyskeepawake - prevent idle system sleep when any tty (e.g. remote login session) is 'active'. A tty is 'inactive' only when its idle time exceeds the system sleep timer.

Yes; if I've got an ssh connection open to a remote host, for a command-line, I do indeed not want it to sleep - _except_ here's the new (to me) part:

ttyskeepawake is apparently for _any_ TTY - _not_ just those open via Terminal, as one might think; see the output, as a result of turning ttyskeepawake off:

Warning: This option disables TCP Keep Alive mechanism when sytem is sleeping. This will result in some critical features like 'Find My Mac' not to function properly.

Example command to turn it off (for all power sources):

pmset -a tcpkeepalive 0

I'll try this over some time, to see if my MacBook finally does stay asleep, safely...

Feedback / suggestions welcome!

3/12 update: Apparently the "sudo pmset -a standbydelay 259200" setting must be performed _while on battery_; if done while on AC / wall power, the on-battery setting is not affected. :/

- - -

Other settings I use (each on a single line), to try to make it sleep quickly, _and_ stay asleep:

sudo pmset -a hibernatemode 0 # do NOT use Safe Sleep; saving time to sleep (and save disk space), by not writing /var/vm/sleepimage

sudo pmset -a standbydelay 259200 # 3 days of sleep, before moving from sleep to hibernate - MUST BE DONE WHILE ON BATTERY

sudo nvram boot-args="darkwake=0" # disable "Dark Wake" (requires reboot)

sudo pmset -a womp 0 # disable "Wake On Magic Packet" (AKA "Wake-on-LAN", AKA "Wake for Wi-Fi Network Access" on recent MacBooks; also seen in the Energy Saver pane in System Preferences)

Also: A command line that I use, to see what the pmset log has, since the most recent sleep event:

telltale='Entering Sleep state'; lastSleep=$(pmset -g log | fgrep -i -e "$telltale" | tail -1 | sed "s/${telltale}.*/$telltale/"); if [ "$lastSleep" ] ; then pmset -g log | sed -n "/$lastSleep/,\$p"; else echo '[WARN: Did not find a Sleep event]'; fi

shards mvgfr: geek