config/VirtualBox_VMs/vegbiendev/README.TXT: to retrieve the original contents of the backup from the VM: added steps to restore the correct VM snapshot

config/VirtualBox_VMs/vegbiendev/README.TXT: also generate list of all the files whose permissions were changed since the backup, but which are extracted with their changed permissions instead of their original ones in the backup

config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to retrieve the original contents of the backup from the VM

fix: /README.TXT: to back up vegbiendev: also back up /home/aaronmk/bien/ (instead of just symlinking to the local copy), since this can be done space-efficiently with hardlinks. this ensures that the vegbiendev backup will not be modified when the local copy of bien/ is.

lib/csvs.py: JsonReader: factored out row-dict-to-list into new row_dict_to_list_reader so that JSON-specific preprocessing is kept separate from the row format translation

lib/csvs.py: added MultiFilter, which enables applying multiple filters by nesting

lib/tnrs.py: single_tnrs_request(): JSON mode: implemented output of JSON data

lib/tnrs.py: single_tnrs_request(): factored out wrapping in TnrsOutputStream, since this is done for both modes

fix: lib/tnrs.py: JSON mode: TSV export columns: need to translate these to JSON column names before they can be used with the JSON data

lib/csvs.py: added JsonReader, which reads parsed JSON data as row tuples

lib/csvs.py: added row_dict_to_list(), which translates a CSV dict-based row to a list-based one

lib/csvs.py: RowNumFilter: added support for filtering the header row as well

lib/csvs.py: ColInsertFilter: added support for filtering the header row as well

lib/csvs.py: InputRewriter: documented that this is also a stream (in addition to inheriting from StreamFilter)

bugfix: lib/csvs.py: InputRewriter: accept a reader, as would be expected, instead of a custom stream whose lines are tuples

fix: lib/sql_io.py: append_csv(): use new csvs.ProgressInputFilter instead of streams.ProgressInputStream(csvs.StreamFilter(__)), so that the input to csvs.InputRewriter is a reader, not a stream. this avoids the need for csvs.InputRewriter to accept a stream whose lines are tuples, instead of the expected reader.

bugfix: inputs/input.Makefile: %/install: $(exportHeader) must come before postprocess because postprocess renames columns

exports/: svn:ignore: added *.gz

lib/csvs.py: added ProgressInputFilter, analogous to streams.ProgressInputStream

lib/sql_io.py: added commented-out debug statement used to troubleshoot copy_expert() errors

lib/dicts.py: added pair_keys(), pair_values()

bugfix: lib/streams.py: CaptureStream: end_idx must also be > start_idx

bugfix: inputs/input.Makefile: $(import_install_): need `set -o pipefail` to enable errexit

/README.TXT: to backup files not in Time Machine: don't need to review diff because command is unidirectional

fix: /README.TXT: to back up the local machine's hard drive: "repeat until only minimal changes" should refer to the first sync command

inputs/.geoscrub/geoscrub_output/run: documented postprocess() rm=1 runtime (6 min)

lib/tnrs.py: single_tnrs_request(): use_tnrs_export=False: need to obtain export columns

lib/csvs.py: added header(stream)

fix: lib/tnrs.py: single_tnrs_request(): need to `assert name_ct >= 1`, because with no names, TNRS hangs indefinitely

bin/tnrs_client: added env var to configure use_tnrs_export

/README.TXT: to back up vegbiendev: use inplace=1 to speed stopping and resuming transfer

fix: /README.TXT: to back up the local machine's hard drive: removed --extended-attributes (after initial sync) because rsync apparently has to visit every file for this

fix: /README.TXT: to back up the local machine's hard drive: also need --extended-attributes

/README.TXT: to back up the local machine's hard drive: removed --delete-before now that that partition has been expanded

fix: /README.TXT: to back up vegbiendev: exclude /var/lib/mysql.bak,postgresql.bak because the local machine doesn't need 2 copies of this information

/README.TXT: to back up vegbiendev: removed no longer needed exclude of Dropbox subdir backup

fix: /README.TXT: to back up vegbiendev: also need to do steps under Maintenance > "to synchronize vegbiendev, jupiter, and your local machine" because /home/aaronmk/bien is not synced here

bugfix: /README.TXT: to back up vegbiendev: need `overwrite=1`

/README.TXT: to back up vegbiendev: removed no longer needed exclude of Dropbox subdir backup

/README.TXT: to back up the version history: don't also need this on vegbiendev because it's already on jupiter and the local machine

bugfix: /README.TXT: to back up vegbiendev: need to include Postgres config files

/README.TXT: to back up the local machine's hard drive: don't back up temp files: added /.fseventsd/

fix: /README.TXT: to back up the local machine's hard drive: initial runtime: use range instead because some of the later runtime might have been from the same files

/README.TXT: to back up the local machine's hard drive: updated initial runtime to include additional transferred files (17 h)

fix: /README.TXT: to back up the local machine's hard drive: need to use --delete-before because the backup partition is near capacity

/README.TXT: to back up the local machine's hard drive: don't back up temp files such as /private/var/vm/*

fix: /README.TXT: to back up the local machine's hard drive: back up most Dropbox/Postgres files before stopping processes, to minimize downtime

bugfix: /README.TXT: to back up the local machine's hard drive: can't use ~ with --exclude

fix: inputs/.geoscrub/geoscrub_output/postprocess.sql: map_geovalidity(): unscrubbable names should actually be geo*in*valid, not geovalid=NULL, according to Brad

/README.TXT: to back up the local machine's hard drive: back up the non-Dropbox, non-Postgres files separately to minimize the Dropbox and Postgres downtime

/README.TXT: to back up the vegbiendev databases: don't need to review diff for these as it's always unidirectional

/README.TXT: added instructions to back up vegbiendev

fix: /README.TXT: to back up the local machine's hard drive: also need to repeat backup command until only minimal changes

/README.TXT: to back up the local machine's hard drive: added step to stop Postgres

bugfix: /README.TXT: to back up the local machine's hard drive: also need to stop Dropbox

/README.TXT: to back up the local machine's settings: added step to remove .DS_Store

fix: /README.TXT: to back up the local machine's settings: Dropbox: shoudl not run with `del=`, because the backup should be an exact replica

backups/TNRS.*: removed no longer needed old TNRS backups, which are part of the respective full-database backups in any case

added config/phpMyAdmin/ symlink to schemas/VegCore/phpMyAdmin/

bugfix: lib/sh/archives.sh: compress(): don't include dir prefix in zip archive

lib/sh/util.sh: cd(): use echo_run instead of a manual echo_cmd call

fix: lib/sh/util.sh: cd(): indent after running cd rather than before

lib/sh/util.sh: cd(): support rebasing path vars for the new dir

bugfix: lib/sh/archives.sh: compress(): need to use zip's path syntax to avoid the file in the archive being named "-"

lib/tnrs.py: added option to avoid using TNRS's TSV export feature, which currently returns incorrect selected matches (vegpath.org/issues/943). this has been implemented up through the GWT/JSON decoding.

lib/tnrs.py: added gwt_decode()

lib/strings.py: added unesc_quotes() and helper functions

lib/strings.py: added json_decode()

/README.TXT: To re-run geoscrubbing: updated runtimes

exports/*_GBIF.csv.run: documented compress_() runtime (20 min-1 h)

lib/runscripts/extract.run: export_(): also compress created file

lib/sh/archives.sh: added compress(), expand(), which handle compression of individual files

bugfix: inputs/input.Makefile: sql/install: ";" for commands inside $(if) blocks need to be inside the $(if) block, too, because otherwise there will be dangling ";" without a statement (bash does not support empty statements containing just ";")

/README.TXT: Full database import: converted database commands to command-line commands to make them easier to run

/README.TXT: Full database import: converted database commands to command-line commands to make them easier to run

web/links/index.htm: updated to Firefox bookmarks: added instructions for how to enable automatic restart on power loss for the UPS (which isn't accessible in the GUI)

fix: schemas/util.sql: contained_within_approx(point geocoord, region postgis.geography): use util.geography() instead of implicit cast to suppress "Coordinate values were coerced into range [-180 -90, 180 90] for GEOGRAPHY" NOTICEs

schemas/util.sql: added geography(util.geocoord), which suppresses "Coordinate values were coerced into range [-180 -90, 180 90] for GEOGRAPHY" NOTICEs

exports/native_status_resolver.csv.run: updated export_() runtime (5 min, now that we're using the narrower New World criterion)

fix: schemas/public_.sql: native_status_resolver: don't include rows with New World coordinates that don't also have New World country names, since the NSR only uses the country name

schemas/public_.sql: native_status_resolver: removed rows with is_geovalid NULL, at Brad's request. note that this removes valid rows with standardized country names.

exports/native_status_resolver.csv.run: updated export_() runtime (30 min)

fix: schemas/public_.sql: native_status_resolver: added country IS NOT NULL filter requested by Brad

fix: schemas/public_.sql: native_status_resolver: remove the id because this prevents SELECT DISTINCT from having the desired effect. instead, the results will be joined back using the other columns.

exports/native_status_resolver.csv.run: upload_(): documented runtime (2.5 min)

bugfix: exports/native_status_resolver.csv.run: upload_(): $live must be exported

exports/native_status_resolver.csv.run: upload_(): use `live=1` instead for consistency with other invocations of put

fix: exports/native_status_resolver.csv.run: upload_(): need `l=1`

exports/native_status_resolver.csv.run: documented export_() runtime (45 min)

exports/native_status_resolver.csv.run: added upload_() to get the file onto nimoy

added exports/native_status_resolver.csv.run

schemas/public_.sql: added native_status_resolver view, requested by Brad (wiki.vegpath.org/Data_requests)

inputs/publishable datasources.xlsx: udpated

lib/tnrs.py: documentation about output of the retrieve step: added that this is also unusable because the array does not contain all the columns and contains no column names

removed no longer used web/BIEN3/Redmine/main/. use Redmine/!__ instead.

web/BIEN3/Redmine/issues/.htaccess: perform .. redirects using new ! prefix

web/BIEN3/Redmine/.htaccess: enable redirects that avoid using a subdir's .htaccess

web/BIEN3/Redmine/wiki/.htaccess: removed no longer needed ignore_fs, since the .htaccess does not have RewriteRules that would need this in a RewriteCond

web/BIEN3/Redmine/issues/.htaccess: main issues page: added default filter conditions

bugfix: web/BIEN3/Redmine/issues/.htaccess: need to redirect to separate URL for individual issues, because they are not located under the issues/ subdir in Redmine