lib/sh/util.sh: added sys_cmd(), which runs a system command and allows running a system command of the same name as the script
lib/sh/util.sh: added echo_builtin()
inputs/.rsync_ignore: test*.xml: turn on syncing again, but always treat the local side of the sync (starscream or vegbiendev) as the authoritative copy since they are the machines the tests can be run on
/.rsync_ignore: temp files: hide them on upload so that they are never synced to jupiter. hiding is different than unidirectionally exclude'ing them, because it also causes them to be deleted on the destination if they were uploaded in previous syncs.
inputs/VegBIEN/TWiki/.rsync_ignore: /**: turn syncing back on, but only allow it unidirectionally from vegbiendev->jupiter->starscream to avoid clobbering the live site or the jupiter backup. this is probably the only dir whose authoritative copy is always on vegbiendev. for all other dirs, edits can be made wherever convenient, so no copy is authoritative and no sync directions need to be restricted.
/README.TXT: Maintenance: synchronization: fixed whitespace
inputs/.rsync_ignore: install.log.sql: only exclude this on starscream (the local machine), using new machine-specific .rsync_filters, so that vegbiendev's copies of this will be backed up
lib/sh/sync.sh: upload(): .rsync_filter: also support machine-specific filters, for cases when different machines produce the same file (e.g. a log file) but only one machine's copy should be backed up
/README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: removed filters that are now handled by .rsync_ignores
added inputs/GBIF/_src/.rsync_filter.upload,download to prevent old versions of GBIFPortalDB-*.dump.gz from being downloaded to the local machine, while keeping them on jupiter. this avoids the need to store these files in ~/Documents/BIEN/large_files/ with symlinks from inputs/GBIF/_src/ to exclude them from the sync.
bugfix: /README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: sync ~/Dropbox/svn/ (the no-unversioned-files working copy) separately from the rest of the files, because .svn/ is now excluded by /.rsync_ignore, so that `svn up` needs to be used to keep the .svn/ dirs in sync. note that .svn/ should generally not be synced between machines, because they may use incompatible versions of the svn working copy format.
/README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: use new bin/sync_upload (with $sync_remote_subdir) so that per-dir .rsync_ignores are processed, and to use the default $sync_remote_url
lib/sh/local.sh: $sync_remote_url: allow user to override just the sync subdir (not the whole URL) in $sync_remote_subdir. this is useful e.g. for backing up the Mac's files to jupiter.
/README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: use new bin/sync_upload instead of specifying all the filter patterns manually. this replaces several `put` commands with various filters with just a bin/sync_upload each on vegbiendev and your machine (in overwrite=1 mode to force a complete sync).
bugfix: backups/.rsync_filter.download: need to prevent existing backups from being deleted on the local side, too, by changing hide patterns to exclude
lib/sh/sync.sh: upload(): make put's $subpath option relative to the currdir instead, like the --include paths. note that $subpath unfortunately can't be used in subdirs at this point because it will cause rsync to ignore the .rsync_ignores and .rsync_filters in parent dirs, including the essential .rsync_ignore in the sync root dir.
/README.TXT: removed unnecessary `env` before kw params, which are treated as such whenever they appear before a command name
bugfix: /README.TXT: updated `make backups/download` to `make backups/<file>/download`
backups/Makefile: upload: use bin/sync_upload
inputs/Makefile: download-logs: use bin/sync_upload like upload/download
bugfix: /README.TXT: `make inputs/upload`, `make inputs/download`: added live=1 so that the sync operation runs rather than previewing what will be synced. removed test=1 because this flag is not used by put.
bugfix: inputs/Makefile: upload, download: need to exclude files in .rsync_ignore, so that large local-only files, such as inputs/GBIF/raw_occurrence_record_plants/table*.tsv, do not have to be synced before `make inputs/upload` can complete (the corresponding .gz gets extracted instead); and deleted temp files in inputs/VegBIEN/TWiki/, such as active sessions, are not added back to the live copy on vegbiendev. previously, fixing this required extracting the rsync command run by `make inputs/upload`, etc. and manually editing it to exclude the applicable .rsync_ignore files, each time `make inputs/upload`, etc. was run (including before every column-based import).
bugfix: bin/make: need to leave bin/, ~/bin/ in the PATH when running make nonrecursively, so that commands invoked by it which are located in these dirs (e.g. put, which will be used by `make inputs/upload`) can still be found. this requires using command()'s new nonrecursive=1 flag instead of running no_PATH_recursion, so that no_PATH_recursion() only affects the resolution of the command path, but does not propagate the filtered PATH to the invoked command itself.
lib/sh/util.sh: command(): added nonrecursive=1 flag, which uses cmd2abs_path to run an external command nonrecursively
lib/sh/util.sh: added cmd2abs_path, which makes the command in $1 nonrecursive
bugfix: lib/sh/util.sh: PATH_rm(): also need to remove adjacent occurrences of the same path (or occurrences which become adjacent when other paths are removed), which :...: matching wasn't doing because the trailing : is consumed, preventing it from being matched at the beginning of the next path. since unlike filesystem paths with /, it is not necessary for a match to span multiple :-separated sections, we can just use new split() to split apart the PATH into an array of paths, filter each path, and join() them back together.
lib/sh/util.sh: added split()
lib/sh/util.sh: auto-echo common external commands: added `which`
lib/sh/util.sh: auto-echo common external commands: use simpler echo_run instead of command since logging handling is not needed
added backups/vegbien.r9897.backup.md5
lib/sh/sync.sh: upload(): documented that each --include path is relative to the currdir, not the root dir of the upload ($local_dir). this feature, although previously unintended, is actually better because the user can change to a subdir of the root dir and specify upload paths relative to the dir they are in. however, when invoking upload() from a script with --include paths specified, this means you need to use an absolute path (e.g. "$(dirname "${BASH_SOURCE0}")"/...; or the value that will become $local_dir, which for sync_upload() is $root_dir).
backups/.rsync_ignore: replaced with .rsync_filter.upload to allow uploading new backups but not deleting existing backups if they don't exist on the local (rsync-invoking) side; and .rsync_filter.download to avoid downloading backups to the local side. this allows storing older backups just on jupiter, where there is much more disk space. note that this change must be made on the remote side (jupiter) for it to be effective, because these are remote-side rules and are only processed by the remote-side rsync instance.
lib/sh/sync.sh: upload(): use directional .rsync_filter to supplement .rsync_ignore with all kinds of --filter rules. separate .rsync_filters are needed for the upload (swap=) and download (swap=1) directions because the sender and the receiver are reversed, causing asymmetric rules like protect/hide to change meaning.
updated backups/TNRS.backup.md5
added backups/TNRS.2013-6-17.backup.md5, TNRS.2013-6-22.backup.md5
/README.TXT: Backups: TNRS cache: Back up/Restore: added runtimes (3 min/5.5 min)
lib/sh/sync.sh: upload(): usage: documented put's swap=1 flag, which downloads instead of uploads
added inputs/GBIF/raw_occurrence_record_plants/.rsync_ignore with filters that have previously needed to be manually added whenever `make inputs/upload` was run
added inputs/GBIF/_MySQL/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.
added inputs/VegBIEN/TWiki/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.
added inputs/.rsync_ignore with filters from inputs/Makefile $(rsyncSrcs). these filters will now be used with bin/sync_upload in addition to `make inputs/upload`.
added bin/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.
added backups/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.
/.rsync_ignore: added *.pyc
added /.rsync_ignore with filters from lib/common.Makefile $(rsync). these filters will now be used with bin/sync_upload in addition to `make inputs/upload`.
lib/sh/sync.sh: upload(): use --exclude filters from per-dir .rsync_ignore. note that --exclude-from can't be used for this, because it is relative to the currdir, not the rsync root, and therefore also requires the .rsync_ignore to exist rather than using it only if it exists.
bin/tnrs_db: documented total runtime (10 days)
bin/tnrs_db: documented current runtime (162 ms/name)
web/links/index.htm: updated to Firefox bookmarks. sorted NCEAS bookmarks to put homepage and support pages first.
/README.TXT: Full database import: To run TNRS, etc. after the main import: clarified that you should only run `export version=<version>` if the import is named something other than public (i.e. it has not yet replaced the previous public schema)
/README.TXT: Full database import: To run TNRS: removed `by_col=1` because by-column mode is not applicable to running TNRS. it is, however, needed when running import_scrub (i.e. `make inputs/<datasrc>/reimport_scrub by_col=1`).
inputs/.TNRS/schema.sql: tnrs: vegbiendev update steps: added `make backups/TNRS.backup-remake` to back up TNRS before making changes to it. this provides a more recent restore point than the last import in case the changes mess things up. (however, the last import's backup is usually sufficient unless TNRS has been run since then.)
inputs/.TNRS/schema.sql: tnrs_populate_fields(): added VACUUM ANALYZE and runtime (50 s)
inputs/.TNRS/schema.sql: tnrs_populate_fields(): updated runtime (16 min)
schemas/VegBIEN/taxonomy/higherPlantGroup.xlsx.src.txt: added Brad's comment that there are some holes in the Embryophyte subclasses list, and we need to validate it
inputs/.TNRS/schema.sql: tnrs: documented that when changing this table's schema, you must also make the same changes on vegbiendev. included sample util.set_col_types() call with runtime (4 min).
bugfix: inputs/.TNRS/schema.sql: tnrs_populate_fields(): need to schema-qualify invoked functions
bugfix: inputs/.TNRS/schema.sql: tnrs_populate_fields(): Is_homonym: use the *_is_homonym flag for whichever of genus or family (in that order) is NOT NULL, rather than horizontal-ORing potentially NULL values together
bugfix: inputs/.TNRS/schema.sql: family_is_homonym(), genus_is_homonym(): need to return NULL instead of false when input family/genus is NULL. EXISTS does not support this, so STRICT is used to provide this functionality automatically.
inputs/.TNRS/schema.sql: added family_is_homonym(), genus_is_homonym() and use them in tnrs_populate_fields()
inputs/.TNRS/schema.sql: score_ok(): changed to IMMUTABLE and STRICT
inputs/.TNRS/schema.sql: tnrs_populate_fields(): never_homonym: use Author_score threshold to exclude matches that are too fuzzy to confirm the presence of a plant name author
bugfix: inputs/.TNRS/schema.sql: tnrs_populate_fields(): *_is_homonym: also need to check that there was no Author_matched (i.e. that it could be a homonym). Is_homonym: use new never_homonym var.
inputs/.TNRS/schema.sql: tnrs_populate_fields(): updated runtime (18 min)
inputs/import.stats.xls: Updated import times
planning/workflow/bien3_architecture.pptx: stage II: removed Step prefix before stage #, which the other slides don't have
added planning/workflow/bien3_architecture/stages.png
added planning/workflow/bien3_architecture/stage_*.png
added planning/workflow/bien3_architecture.pptx
inputs/.TNRS/schema.sql: tnrs_populate_fields(): when changing this function: UPDATE statement: include TNRS schema since it may not be in the search_path
inputs/.TNRS/schema.sql: tnrs_populate_fields(): Is_plant: also consider homonyms using new family_is_homonym, genus_is_homonym (see wiki.vegpath.org/Result_filtering#taxon_is_plant)
inputs/.TNRS/schema.sql: tnrs: added Is_homonym derived col (uses IRMNG.family_homonym_epithet, genus_homonym_epithet)
schemas/vegbien.sql: re-ran `make schemas/public/reinstall; make schemas/remake` cycle, which apparently changed sort order of statements
/README.TXT: Full database import: disk space check: updated minimum (to 300GB) for new import schema size. note that most of the space (166GB) is indexes, and even of the 87GB of data, only 20GB is from GBIF and 15GB from FIA (so most of it is duplication).
added inputs/IRMNG/*_homonym_epithet/map.csv, etc. (created by */run)
bugfix: inputs/input.Makefile: `%/install %/header.csv: %/create.sql`: in noclobber mode, mark %/header.csv as .PRECIOUS so the existing file won't be deleted if the table already exists (causing an error exit)
bugfix: lib/runscripts/table.run: remake_VegBIEN_mappings(): run yes using piped_cmd() so the SIGPIPE doesn't cause an errexit
added inputs/IRMNG/{genus_homonym_epithet,family_homonym_epithet}/run, which inherit from ../table.run so that load_data() (which runs create.sql) is invoked
added inputs/IRMNG/species_homonyms/new_terms.csv
bugfix: added no-op inputs/IRMNG/Source/run so inputs/IRMNG/run would have something to invoke for it
inputs/IRMNG/run: use lib/runscripts/datasrc_dir.run, which now provides import() and $subdirs
lib/runscripts/datasrc_dir.run: extend import.run and provide an import() implementation that runs all the runscripts for import_order.txt subdirs
lib/csvs.py: sniff(): support single-column spreadsheets by defaulting to the Excel dialect when the delimiter can't be determined
inputs/IRMNG/: added family_homonym_epithet, genus_homonym_epithet lookup tables, which use util.all_same() to filter out internal Plantae homonyms
schemas/util.sql: added all_same() aggregate
schemas/util.sql: added not_empty(anyarray)
schemas/util.sql: added not_null() (usable as an aggregate's FINALFUNC)
bugfix: inputs/IRMNG/import_order.txt: need to specify order so that Source is first
bugfix: inputs/IRMNG/*/map.csv: remapped Authority to scientificNameAuthorship instead of authors (now data_authors <VegCore.vegpath.org?data_authors> for clarity)
inputs/IRMNG/map.csv: updated to scrubbed output names from */map.csv (/map.csv does not currently get scrubbed)
bugfix: inputs/IRMNG/species_homonyms/header.csv, map.csv: reset input columns to DSV (delim-separated values) header. they had gotten changed to the output names in running map.csv with remake=1, causing it to be remade from the (renamed) staging tables.
inputs/input.Makefile: $(_svnFilesGlob): added *Makefile
/README.TXT: `make inputs/{upload,download}`: first run with test=1 to see what the diffs will be
added inputs/IRMNG/, including runscripts to download the names. this is now the 2nd datasource after GBIF to use runscripts, and the 3rd after FIA/GBIF to use new-style import.
inputs/input.Makefile: $(_svnFilesGlob): added *run (runscripts)
lib/runscripts/table.run: import(): also run remake_VegBIEN_mappings() to accept the test output. this function was previously unused, but was left in for future use when lib/import.sh was translated to lib/runscripts/table.run (it was used in its import.sh form in inputs/FIA/occurrence_all/import).
bugfix: lib/runscripts/table.run: remake_VegBIEN_mappings(): need to change to $top_dir before running `rm header.csv map.csv`
