fix: /README.TXT: to back up the local machine's hard drive: don't need to pause Dropbox, since it will be reinitialized for the new partition UUID anyway

planning/meetings/BIEN conference call availability.xlsx: updated

bugfix: /README.TXT: to back up the local machine's hard drive: first need to resize the BIEN3.**SAVE** partition so there is enough space

bugfix: /README.TXT: to backup files not in Time Machine: need to use `brew services _ postgresql` instead of `launchctl _ ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist`

bugfix: /README.TXT: to back up the local machine's hard drive: need to use `brew services _ postgresql` instead of `launchctl _ ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist`

bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to retrieve the contents of the VM: can't use inplace=1 optimization because this messes up the shared hardlinks used by ~/Documents/BIEN/vegbiendev/

fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to retrieve the contents of the VM: don't need to boot the VM because those snapshots are already booted

bugfix: /README.TXT: to back up vegbiendev: can't use inplace=1 optimization because this messes up the shared hardlinks used by ~/Documents/BIEN/vegbiendev.2014-2-2_1-07-32PT*/

bugfix: /README.TXT: to back up vegbiendev: also need to lock hardlinked files at end

lib/sh/util.sh: added errexit_sig()

lib/sh/util.sh: sig_e(): use new sig_num()

lib/sh/util.sh: added sig_num()

bin/import_all: `make -s`: removed no longer needed output_data=1

bugfix: lib/sh/make.sh: make() wrapper: need to set output_data mode if -s is set, to match the behavior of the underlying make

bugfix: bin/import_all: don't disable errexit because this prevents the program from being Ctrl-C'd. this functionality is no longer needed now that the README.TXT instructs to run bin/import_all in a subshell.

bin/import_all: removed functionality now provided by util.run

bin/import_all: converted to a runscript so it can use runscript functionality

/README.TXT: Installation: `make install`: added alternative command to install a complete DB with all the datasources (instead of a blank DB)

inputs/bien_web/observation/map.csv: updated

bugfix: inputs/Makefile: install: need to use $(wildcard+suffix) to support Linux

lib/common.Makefile: added $(wildcard+suffix), needed because on Linux, builtin $(wildcard) doesn't handle suffixes that don't exist

inputs/Makefile: install: support $(all) mode like for download

inputs/Makefile: download: don't download log files, to speed up installation on a new machine

inputs/Makefile: download: don't download all the inputs by default, to speed up installation on a new machine. instead, enable this with the $(all) flag.

bugfix: /README.TXT: Full database import: need to unlock hardlinked files

/README.TXT: back up Postgres: backups/pg_snapshot: documented runtime when queries have been run (1 h)

fix: /README.TXT: Full database import: also need to run `unset TMOUT` in the outer `screen` shell

fix: /README.TXT: back up Postgres: needs to be run in `screen`

/README.TXT: to synchronize vegbiendev, jupiter, and your local machine: update mtimes in Dropbox: preserve uncommitted changes

fix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: update mtimes in Dropbox: don't update perms because this would incorrectly transfer the hardlink lock status

fix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: TWiki: need to fix_perms after running this to reset perms on hardlinks

/README.TXT: syncing: use del instead of overwrite where possible, to avoid overwriting newer files

/README.TXT: Full database import: should also snapshot Postgres before starting import

bugfix: inputs/import.stats.xls: need to exclude postprocessing from the ms/row and Change formulas, also for the "<2014-2-2" tab

inputs/import.stats.xls: updated import times

inputs/import.stats.xls: updated import times

/README.TXT: record the import times: documented how to update the Postprocessing times

bugfix: /README.TXT: globs that list all the datasources: this was incorrectly excluding U, so fixed to match bin/with_all $hidden_srcs

fix: inputs/VegBank/project/: added _no_import because this is imported as part of taxon_observation.**

bugfix: inputs/import.stats.xls: need to exclude postprocessing from the ms/row and Change formulas. removed deleted rows that don't apply to the most recent imports. updated runtime formulas to match bin/import_all.

bugfix: inputs/import.stats.xls: restored missing formatting for multi-day times. prepped tabs to have new import stats data added.

fix: /README.TXT: record the import times: need to first click the "current" tab

/README.TXT: To re-run geoscrubbing: updated runtimes and revision #s

bugfix: /README.TXT: To re-run geoscrubbing: don't close subshell because it closes automatically (this would instead exit `screen`)

/README.TXT: To re-run geoscrubbing: updated runtimes

_license/VegBank_schema/applies_to.txt: added exports/sPlot/BIEN-sPLot-Skype-finalNotes.docx

added exports/sPlot/BIEN-sPLot-Skype-finalNotes.docx from Bob

/README.TXT: To re-run geoscrubbing: updated runtimes

bugfix: /README.TXT: To re-run geoscrubbing: "press Enter" and `unset TMOUT` steps are reversed

bugfix: schemas/public_.sql: geoscrub_input_new: need to join on the ARRAY[] of each column, to compare NULLs properly

bugfix: lib/sh/util.sh: become_sudo don't continue running function after self_sudo finishes

/README.TXT: datasource removal: linked to runtimes on the wiki

bugfix: inputs/input.Makefile: %/map.csv: need to save it if errors occur in unmapped_terms.csv, new_terms.csv

fix: inputs/FIA/PLOT/map.csv: added plotArea_ha, requested by Brian McGill

fix: inputs/FIA/PLOT/map.csv: added plotArea_ha, requested by Brian McGill

added backups/old_backups.txt

added backups/TNRS.2014-6-2.backup.md5

_license/non-open-source/applies_to.txt: clarified that the things in this list are not embargoed, the issue is just that they are not formally open-source

schemas/public_.sql: view_full_occurrence_individual_view: join to taxonoccurrence: can use a USING join for this because plot.** only exports one locationevent_id column

inputs/Cyrille_traits/trait_observation/postprocess.sql: accessRights index: added vegbiendev runtime

fix: inputs/Cyrille_traits/trait_observation/postprocess.sql: need to remove private data, as clarified by Cyrille

inputs/publishable datasources.xlsx: updated: Cyrille traits: updated with Cyrille's clarifications on publishability

bugfix: lib/sh/sync.sh, db.sh: *_snapshot() commands that run multiple sudo commands: use become_sudo so that the program won't hang at later sudo prompts. for pg_snapshot() with pg_start_backup(), this also prevents the backup from staying open at the end waiting on a sudo prompt, causing excess WAL logs to accrue.

lib/sh/util.sh: become_sudo: documented that this is for funcs that run multiple sudo commands

lib/sh/util.sh: $sudo_cmd_wrapper: use esc() so this does not need to be embedded with [@] array notation

bugfix: lib/runscripts/util.run: $wrap_fn: this always needs to contain `is_runscript=1` because $is_runscript is not (and cannot be) exported

bugfix: lib/sh/util.sh: sudo(): when don't need sudo: do need `command` to have effects of invoking external command

fix: lib/sh/db.sh: pg_snapshot(): first need to run pg_stop_backup() in case a backup was not stopped cleanly

fix: lib/sh/db.sh: pg_stop_backup(): ignore benign "a backup is not in progress" error

lib/sh/util.sh: sudo(): only run sudo if needed

lib/sh/util.sh: sudo(): factored PATH-preserving code out into separate wrapper var that can also be used separately, eg. with `sudo -u`

lib/sh/util.sh: sudo(): moved addition of -E (which should always happen) to separate, overridden sudo() function

lib/sh/util.sh: moved -E to is_callable section because it just applies to that

fix: inputs/input.Makefile: $(svnFilesGlob): only svn:ignore *.log in the top-level dir

added inputs/XAL/Specimen/data.csv (empty template)

fix: inputs/input.Makefile: add!: verify/: also svn:ignore .zip

fix: inputs/input.Makefile: add!: verify/: also svn:ignore .zip

lib/sh/util.sh: added become_sudo alias

lib/sh/util.sh: added $is_root

inputs/publishable datasources.xlsx: updated: replaced BIEN2 traits with Cyrille traits

exports/view_full_occurrence_CVS.csv.run: documented runtime (2 min for 5000 rows)

fix: inputs/publishable datasources.xlsx: updated: conditions of use: Canadensys sources: these should actually be marked as no restrictions, in contrast to Brian E's earlier annotations, because they are public domain: the citation is requested, but not legally required

inputs/publishable datasources.xlsx: updated

inputs/publishable datasources.xlsx: updated: conditions of use: indicated which datasources have no restrictions

web/BIEN3/Redmine/issues/.htaccess: dest: decoded chars that don't need to be escaped, for clarity

bugfix: web/BIEN3/Redmine/issues/.htaccess: need to store dest in separate var because % is treated specially in the replacement str

inputs/CVS/^taxon_observation.**.sample/test.xml.ref: updated

fix: inputs/CVS/plantConcept_/header.csv: regenerated after fixing the postprocess/cleanup ordering bug (r14827), which had caused header.csv to be incorrectly generated after renames in postprocess.sql were applied

web/BIEN3/Redmine/issues/.htaccess: main page: sort by priority

planning/meetings/BIEN conference call availability.xlsx: updated

planning/meetings/BIEN conference call availability.xlsx: updated

planning/meetings/BIEN conference call availability.xlsx: updated: added and populated fall 2014 tab

bugfix: inputs/input.Makefile: postprocess must be run after cleanup rather than before because it depends on the cleanup having been performed.

this bug was not previously detected because this is only a problem when refreshing a datasource to data in the same format: this would attempt to run an existing postprocess.sql, out of order, instead of starting with no postprocess.sql as we usually do....

lib/sql_io.py: cleanup_table(): trim(): documented that this also converts character varying fields to text

bugfix: inputs/input.Makefile: $(dbExports): also need to put data.sql before clean_up.sql, etc. previously, this ordering had to be done by naming clean_up.sql, etc so they would sort after data.sql alphabetically, but it can be confusing to have to remember to do this. this fixes a bug in the CVS refresh where cvs.~.clean_up.sql was being run before data.sql, causing some private columns to have been deleted before the data was imported into the tables, creating a column mismatch error.

/README.TXT: To re-run geoscrubbing: re-create public-schema views that were cascadingly deleted: listed which views these currently are

/README.TXT: To re-run geoscrubbing: updated runtimes

bugfix: lib/sql_io.py: null_strs_str_default: removed "NA" because this is the abbr for a Spanish province (Navarra). this fixes the 2nd bug of #955, geovalidation duplicated rows.

lib/sql_io.py: cleanup_table(): debug-print null_strs

lib/sql_io.py: null_strs: made it customizable from an env var, since the same list of null_strs doesn't work for all datasources (see #957)