backups/*retention_policy*: on jupiter: changed to avoid retaining backups further back, as these consume significant disk space on jupiter and are not useful for anything

fix: lib/sh/local.sh: $sync_remote_url: need $USER so user can be overridden when running as root

fix: /README.TXT: to back up the local machine's settings: to jupiter: need `sudo -E` for new Documents/BIEN/vegbiendev*/

/README.TXT: backups of ~/.dropbox/: just pause and resume Dropbox instead of quitting and restarting it

/README.TXT: to back up the local machine's settings: to Dropbox: exclude gmvault-db/ to save space in Dropbox

/README.TXT: to back up the local machine's settings: don't need to exclude ~/Library/Thunderbird/Profiles/9oo8rcyn.default/global-messages-db.sqlite because the disk that was corrupting this file has been replaced (#907)

/README.TXT: to back up e-mails: don't need to also sync aaronmk@nceas.ucsb.edu because these e-mails are also in aaronmk.nceas@gmail.com (auto-forwarded)

/README.TXT: Notes on system stability: removed no longer applicable warning about system upgrades, which is now incorrect because the disk space overrun bug (#887) was found not to have been caused by a system upgrade

web/links/index.htm: updated to Firefox bookmarks: Roundtable: added links for Roundtable on "obstacles faced by researchers who reuse, share and manage data, and strategies for overcoming them"

lib/Firefox_bookmarks.reformat.csv: label page's self-description as such: also support quotations enclosed in '

fix: exports/native_status_resolver.csv.run: added `users_by_name=1` to preserve the file group

added inputs/VegBank/run.call_graph.log

inputs/VegBank/run.log: updated for echo_vars() changes. the PG* vars, which contain important information, will now not need to be filtered out.

lib/sh/util.sh: echo_vars(): merge repeated flags so there aren't flags in between the vars (which is also not valid declare syntax)

lib/sh/db.sh: pg_cmd(): log vars on same line to avoid clutter

lib/sh/util.sh: echo_vars(): put all the vars on the same line so they don't clutter up the call graph generated at the default verbosity

web/links/index.htm: updated to Firefox bookmarks: Mac: added link for VNC client. extended attributes: added link for chflags.

added planning/workflow/staging_tables_installation_for_SQL_datasource.odg.src.log

added inputs/VegBank/run.log

fix: inputs/input.Makefile: $(svnFilesGlob): *.log should be in both the subdirs and the main dir

inputs/input.Makefile: $(svnFilesGlob): *.log

inputs/Makefile: install: install an empty VegBIEN schema instead of all the datasources, at Mark's request. this enables loading just a single datasource.

schemas/VegBIEN/data_dictionary/VegBIEN data dictionary.xlsx: updated

bugfix: schemas/public_.sql: view_full_occurrence_individual_view and related views: synced to data dictionary spreadsheet, which adds back the links to the definitions (which used to be part of the column name itself)

fix: schemas/public_.sql: analytical_plot, analytical_specimen: updated column names to be the same as analytical_stem, which these are a subset of

/README.TXT: to synchronize vegbiendev, jupiter, and your local machine: avoid extraneous diffs when rsyncing: clarified the machines that the command should be run on

web/links/index.htm: updated to Firefox bookmarks: removed broken favicons

web/links/index.htm: updated to Firefox bookmarks: updated favicons

fix: /README.TXT: to backup files not in Time Machine: need to use 2 TB external hard drive instead of Time Machine drive because Time Machine drive does not have ~/Documents/BIEN/ in a location where it can be hardlinked against

web/links/index.htm: updated to Firefox bookmarks: categorized uncategorized bookmarks

web/links/index.htm: updated to Firefox bookmarks: updated favicons

web/links/index.htm: updated to Firefox bookmarks: local machine phpPgAdmin: removed this so the Mac won't get woken up on network access whenever someone opens the links page, which attempts to load the favicon from the local machine. the previous solution of manually deleting the favicon (r13406) doesn't work because the favicon will just get re-added whenever this bookmark is visited.

web/links/index.htm: updated to Firefox bookmarks: find: added instructions for searching by <>, not just =

/README.TXT: Datasource setup: For MS Access databases: added that one should use the settings in the associated .ini file where available

/README.TXT: Datasource setup: For MS Access databases: program link: added page subsections

/README.TXT: to backup files not in Time Machine: note that Time Machine dereferences hard links: added commands documenting that this is the case

fix: /README.TXT: to backup files not in Time Machine: on first run, create parent dirs: added mkdir for Postgres

bugfix: /README.TXT: to backup files not in Time Machine: on first run, create parent dirs: mkdir: need sudo

/README.TXT: to backup files not in Time Machine: moved to root/ subdir to group the multiple top-level dirs together

/README.TXT: to backup files not in Time Machine: added the vegbiendev archival backups, which cannot be backed up by Time Machine because it dereferences hard links

/README.TXT: to backup files not in Time Machine: documented why Postgres cannot be backed up by Time Machine

/README.TXT: Single datasource refresh: added steps to place the updated extract and extracted flat file(s)

/README.TXT: Single datasource refresh: connect to vegbiendev first, even though steps before it have their own step to do this

/README.TXT: Single datasource refresh: reimport_scrub: added step to view progress

/README.TXT: Single datasource refresh: moved to top since these steps are performed more often

added planning/workflow/BIEN data workflow-2_bje.png export

planning/meetings/BIEN conference call availability.xlsx: updated

planning/workflow/BIEN data workflow-2_bje.pptx: updated with Martha's changes and changes during conference call

bugfix: web/BIEN3/Redmine/.htaccess: subpath redirect: also redirect dirs, so that empty-subdir main-page redirects (eg. wiki.vegpath.org) work properly

bugfix: web/BIEN3/Redmine/.htaccess: main page should continue to redirect to wiki, not Redmine project page

schemas/public_.sql: *_view: re-ran *_view_modify(), which use the new non-blocking rematerialize_view()

schemas/public_.sql: viewFullOccurrence_*: renamed to view_full_occurrence_* at Brian M's and Martha's request (e-mails from Martha on 2014-8-12 at 17:37PT, and from Brian M on 2014-8-13 at 16:21PT). note that this change has already been made on vegbiendev.

schemas/public_.sql: view_full_occurrence_individual: re-ran view_full_occurrence_individual_view_modify(), which uses the new non-blocking rematerialize_view()

schemas/util.sql: rematerialize_view(): made it non-blocking, so that it would allow full access to the original materialized table during the operation

schemas/util.sql: added identifier_replace()

schemas/util.sql: added relation_replace()

/README.TXT: Single datasource import: renamed to Single datasource refresh since it works on existing datasources

/README.TXT: Single datasource import: also need to reload staging tables

/README.TXT: Single datasource import: added steps to re-run geoscrubbing and back up the vegbiendev database

planning/workflow/BIEN data workflow-2_bje.pptx: fixed text alignment

planning/workflow/BIEN data workflow-2_bje.pptx: answered questions asked in the diagram

added planning/workflow/BIEN data workflow-2_bje.pptx from Martha/Brian E (in Asana)

added inputs/CVS/verify/Review of CVS data in BIEN3.docx

backups/*retention_policy*: added explanations

backups/*retention_policy*: on jupiter: backups further back: removed "if disk space permits" because this is already labeled "optionally"

backups/*retention_policy*: changed to require retaining *.backup of the last 2 successful imports on all machines

backups/*retention_policy*: allow keeping *.backup of the last 2 successful imports on all machines, not just jupiter

: renamed 2TB drive's BIEN3 partition to BIEN3.**SAVE since one might not see the SAVE file in it

: renamed 2TB drive's BIEN3 partition to BIEN3.**SAVE since one might not see the SAVE file in it

/"**DO_NOT_DELETE": renamed to shorter SAVE

added backups/*retention_policies*/ with retention policy files for each partition

backups/README.TXT: renamed to retention_policy to match the naming convention of the retention policy files in the various partitions

/README.TXT: to back up the local machine's hard drive: also exclude *-files indicating the (differing) retention statuses of the partitions involved

lib/tnrs.py single_tnrs_request(), bin/tnrs_client: use_tnrs_export: default to False because this mode uses incorrect selected matches (vegpath.org/issues/943), and the JSON mode that fixes this is now available

bin/tnrs_db: tnrs.tnrs_request() call: explicitly set use_tnrs_export=True so that this continues to work if the default value is changed

bugfix: lib/csvs.py: JsonReader: need to pass col_order to row_dict_to_list_reader

config/VirtualBox_VMs/vegbiendev/README.TXT: ~/Documents/BIEN/vegbiendev.2014-2-2_1-07-32PT.+VirtualBox_changes/: renamed to vegbiendev.2014-2-2_1-07-32PT.VirtualBox/ to make clear that this is the VirtualBox version of vegbiendev

bugfix: lib/tnrs.py: JSON output: need to stringify arrays so they match what is output in TSV-export mode

lib/csvs.py: JsonReader: added support for values that are arrays

lib/csvs.py: MultiFilter: inherit from WrapReader instead of Filter to avoid needing to define a no-op filter_() function

bugfix: lib/csvs.py: row_dict_to_list_reader: need to override next() directly instead of just using Filter, because Filter doesn't support returning multiple rows for one input row (in this case, prepending a header row). this caused the 1st data row to be missing.

lib/csvs.py: Filter: inherit from WrapReader, which separates out the CSV-reader API code

lib/csvs.py: added WrapReader

lib/csvs.py: added Reader

schemas/public_.sql: views that use view_full_occurrence_individual_view: use the view_full_occurrence_individual table instead, now that this is materialized.

planning/meetings/BIEN conference call availability.xlsx: updated

/README.TXT: to back up the local machine's hard drive: renamed backup partition to BIEN3 to make clear what the backup drive contains

fix: /README.TXT: to back up the local machine's hard drive: updated location of `screen` for added commands

/README.TXT: added trailing / on dirs to make clear that they're dirs

config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to configure the VM to support VirtualBox

config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to retrieve the contents of the VM, with the VirtualBox changes added

config/VirtualBox_VMs/vegbiendev/README.TXT: to retrieve the original contents of the backup from the VM: added steps to restore the correct VM snapshot

config/VirtualBox_VMs/vegbiendev/README.TXT: also generate list of all the files whose permissions were changed since the backup, but which are extracted with their changed permissions instead of their original ones in the backup

config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to retrieve the original contents of the backup from the VM

fix: /README.TXT: to back up vegbiendev: also back up /home/aaronmk/bien/ (instead of just symlinking to the local copy), since this can be done space-efficiently with hardlinks. this ensures that the vegbiendev backup will not be modified when the local copy of bien/ is.

lib/csvs.py: JsonReader: factored out row-dict-to-list into new row_dict_to_list_reader so that JSON-specific preprocessing is kept separate from the row format translation

lib/csvs.py: added MultiFilter, which enables applying multiple filters by nesting

lib/tnrs.py: single_tnrs_request(): JSON mode: implemented output of JSON data

lib/tnrs.py: single_tnrs_request(): factored out wrapping in TnrsOutputStream, since this is done for both modes

fix: lib/tnrs.py: JSON mode: TSV export columns: need to translate these to JSON column names before they can be used with the JSON data