lib/tnrs.py: submission_request_template: include GCC in addition to Tropicos, because it provides more synonyms than Tropicos for Asteraceae, and the accepted names still match the Tropicos backbone (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/2013-06-13_conference_call#include-GCC-when-running-TNRS)
inputs/.TNRS/tnrs/tnrs.make: removed no longer needed end time, now that the total runtime is printed
inputs/.TNRS/tnrs/tnrs.make: print the total runtime using `time`
inputs/.TNRS/tnrs/tnrs.make: include the end time in addition to the start time so that the total runtime can be calculated
lib/sh/util.sh: command-specific alternate stdin/stdout/stderr: choice of 40/41/42: added mnemonic that 4 looks like A for Alternate
bugfix: /README.TXT: Full database import: added step to remove any leftover TNRS lockfile. usually, the PID in it would not exist, but sometimes it now refers to a different, active process which blocks tnrs.make.
schemas/vegbien.sql: allow public_ to view lookup tables (cultivated_family_locations, higher_plant_group_nodes)
added backups/TNRS.backup.md5, vegbien.r9459.backup.md5
bugfix: lib/sh/local.sh: sync_upload(): need to use --no-group to prevent the group from being reset to aaronmk upon download from jupiter (which uses group aaronmk instead of bien). use ./fix_perms to set the group of all files to bien. also use --no-owner in case running as root.
lib/sh/sync.sh: removed sync_download(). use swap=1 sync_upload() instead.
lib/sh/sync.sh: removed download(). use swap=1 upload, or swap=1 upload_caller, instead. this avoids having separate upload()/download() pairs for every caller of upload(), because you can instead just set swap=1.
bugfix: lib/sh/sync.sh: upload(): don't kw_params $swap because this unexports it, preventing put from seeing it. instead, use echo_vars to print it.
added bin/sync_upload, a wrapper around sync_upload()
bugfix: lib/sh/sync.sh: upload(): only add `--exclude="**"` if there are --includes. this enables running upload() without paths to upload all files.
lib/sh/sync.sh: upload(): support passing -- options to put, which will not be run through the path->--include processing
bugfix: lib/sh/sync.sh: upload(): added missing `local args=()` initializer
/README.TXT: Full database import: On local machine: added step to do steps under Maintenance > "to synchronize vegbiendev, jupiter, and your local machine", which is needed in addition to `make inputs/upload` since that doesn't handle overwrites or deletions
/README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: added warning that you should pay careful attention to all files that will be deleted or overwritten (as the three machines are often out of sync)
added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.0.preamble.sql
/README.TXT: Full database import: make inputs/{upload,download}: run them first with `test=1` to see what the changes will be
/README.TXT: Full database import: `svn up`: use --force to avoid errors about existing files
mappings/VegCore-VegBIEN.csv: genus->taxonlabel.taxonomicname: filter out genera that contain numbers (using new _filter_genus()), which break TNRS and prevent it from matching any other parts of the name. later, these genera can instead be moved to the end of the name, where TNRS will correctly match them as Unmatched_terms.
bugfix: inputs/VegBIEN/: added _no_import to disable import for this folder, since this is actually just an entry in web/datasources/ with VegPath redirection links, rather than an input to the import process. this fixes "schema "VegBIEN" does not exist" errors generated in `make test`.
inputs/input.Makefile: $(dontImport): also support putting a _no_import file at the top level in the datasource to exclude the entire datasource
bugfix: lib/sh/local.sh: removed make() override, which is no longer needed now that its operations are performed by verbosity_compat(), and which caused errors by setting $verbosity to the invalid value ""
bugfix: lib/sh/util.sh: verbosity_compat(): always use default verbosity (`unset verbosity`) when verbosity == 1, regardless of whether the caller has set $verbosity to the special value "", because $verbosity is supposed to be an integer field and "" is not supported by most functions that use $verbosity. in cases where a util.sh script is invoked, it will set $verbosity back to the default value 1, so this will function as before for util.sh scripts and fix $verbosity for scripts that use a different verbosity system.
added inputs/GBIF/raw_occurrence_record_plants/table.tsv.md5
inputs/GBIF/raw_occurrence_record_plants/test.xml.ref: regenerated. updated for new staging table input columns, which are now the same as the output columns.
bugfix: inputs/input.Makefile: %/VegBIEN.csv: use header from map.csv instead of the new columns, so that source.shortname is set to GBIF instead of VegCore
inputs/input.Makefile: %/VegBIEN.csv: when a runscript is available, instead map the output columns of map.csv to VegBIEN, because the columns have been renamed in the staging table
inputs/GBIF/raw_occurrence_record_plants/VegBIEN.csv: regenerated, which adds row_num input col
lib/sh/util.sh: echo_func(): check can_log at beginning of function, so that the resource-intensive func_loc (which calls `readlink -f`) does not need to be called if echo_cmd would not log anything at the current verbosity
lib/sh/util.sh: echo_func(): removed no longer used $minor flag. use `clog++... echo_func` instead.
lib/sh/util.sh: verbosity_compat(): don't make $verbosity a local var of this function, because then the changes will not be visible to the caller
bugfix: bin/make: use verbosity_compat because some make-invoked commands (e.g. bin/map) don't support verbosity=""
lib/sh/util.sh: command(): command__exec(): use verbosity_compat to support commands that don't support verbosity=""
lib/sh/util.sh: added verbosity_compat(), for use with commands that don't support verbosity=""
bugfix: lib/sh/local.sh: make(): when invoking overridden func, need make__make_sh
bugfix: lib/sh/util.sh: self, self_sys aliases: need to remove any func_override suffix __* from the FUNCNAME
bugfix: inputs/GBIF/import_order.txt, run: updated raw_occurrence_record/ to raw_occurrence_record_plants/
inputs/FIA/occurrence_all/test.xml.ref: update inserted row count
bugfix: bin/make: include local.sh so that its default verbosity-setting make() override will be used
lib/sh/local.sh: added make() override, which uses the default verbosity (i.e. verbosity="") when verbosity == 1. scripts that use lib/sql.py (which uses $verbosity) have different default verbosities, and this default should not be overriden by an env var, unless a higher verbosity has been set.
lib/sh/local.sh: added missing include of make.sh, used by root_make()
schemas/vegbien.sql: added _filter_genus()
inputs/GBIF/raw_occurrence_record_plants/run: import() runtime: specified that this does not include table.tsv.gz/make()
inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: # duplicates: added revision #
inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: documented that there are 4.5 million duplicates (59,998,354 rows before - 55,417,646 rows after = 4,580,708)
inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: added rerun time (~0 thanks to index, so no problem doing the DELETE each time postprocess.sql is run)
*{.sh,run}: use simpler .rel() instead of `. "$(dirname "${BASH_SOURCE0}")"/...` for relative includes
lib/sh/util.sh: added .rel()
schemas/VegCore/VegCore.pg.sql: regenerated from installed schema. a Linux bug that caused constraints to be output in reverse order has now been fixed, as of 12.04.2 (broken in 12.04.1).
bugfix: inputs/GBIF/_MySQL/MySQL_schema, MySQL_data: sed: put {} commands on their own line to work on Mac
lib/sh/util.sh: .(): put echo_func on its own line for clarity
lib/sh/util.sh: .(): added echo_func (2 log_levels up because it's internal)
schemas/VegCore/mk_derived: use new lib/sh/local.sh instead of lib/import.sh (a precursor to util.sh, etc. still used by inputs/FIA/)
bugfix: load_data(): verbosity_min: use verbosity_min='' so that csv2db's default verbosity (3) is used, instead of setting the verbosity directly to 3, which caused the log++ logging output from bin/make to be echoed at verbosity 3, creating cluttered output
lib/sh/util.sh: verbosity_min(): support value '', which sets verbosity=''
bugfix: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: updated column names to match the renamings in map.csv, which are now performed on the staging table itself
lib/sh/util.sh: run_args_cmd(): time the command so that the runtime of the outer runscript target (i.e. the command run from the shell) is printed at the end of the output, like in bin/make
bugfix: inputs/input.Makefile: %/install: don't run $(cleanup) if it has already been run by $(import_install_), so that it doesn't run twice
inputs/input.Makefile: %/postprocess: don't run postprocess.sql if it is supposed to be run by a runscript, because postprocess.sql may then depend on additional steps the runscript runs before it
lib/runscripts/table.run: import(): use self_make on load_data so that the remake status determines whether the table is reinstalled
bugfix: lib/runscripts/mysql.table.run: import(): added missing set_make_vars, needed by self_make
bugfix: lib/runscripts/table.run: load_data(): need to use $_remake instead of $remake when using set_make_vars
lib/runscripts/table.run: added set_make_vars to all make targets so $remake would be propagated appropriately
lib/runscripts/table.run: load_data(): also clobber install log if remaking, because the table will be reinstalled
lib/runscripts/table.run: load_data(): automatically select noclobber mode depending on whether the install log already exists. this removes the need for a separate load_data_first_run() function.
bugfix: lib/runscripts/table.run: load_data(): ignore errors if table already exists
lib/runscripts/table.run: load_data(): use noclobber=1 to avoid overwriting the install log when re-running the install target idempotently. load_data_first_run() is now available to preserve the output in the log on the first run.
inputs/input.Makefile: Staging tables installation: $(logInstall): don't output to the install log if $noclobber flag is set, to prevent overwriting the log when re-running the install target idempotently
bugfix: lib/runscripts/mysql.table.run: import(): move previous versions of table.tsv out of the main dir before loading staging tables, to prevent them from being considered a TSV segment file and prepended to table.tsv
lib/sh/util.sh: added mv2dir(), mv_glob which wrap mv
lib/sh/util.sh: added mkdir alias which adds -p to prevent errors if the dir already exists
lib/sh/util.sh: added wildcard alias, similar to make's $(wildcard) function
bugfix: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: institution_code index: create it idempotently using create_if_not_exists() and an explicit index name, so that a duplicate index doesn't get added each time postprocess.sql is run
lib/sh/local.sh: psql(): don't put util in the search_path because psql scripts now add it themselves if they need it, using `SELECT util.search_path_append(util);`
inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: add util to the search_path so that postprocess.sql will also work when run by inputs/input.Makefile, which only puts the datasource (GBIF) in the search_path
schemas/util.sql: added search_path_append()
schemas/util.sql: added eval() to allow running EXECUTE outside of a function (and to echo the command that is run)
inputs/GBIF/raw_occurrence_record_plants/run: added import() runtime (5 h)
inputs/GBIF/raw_occurrence_record_plants/run: table.tsv.gz/make() runtime: noted that this excludes the upload time
inputs/GBIF/raw_occurrence_record_plants/run: added table.tsv.gz/upload() runtime (15 min)
added lib/runscripts/mysql.table.run (general to all MySQL datasources) and use it in inputs/GBIF/table.run
inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): to view runtime when using `screen`: keys used to scroll: added Ctrl-B/Ctrl-F for page-at-a-time scrolling (there are a lot of pages of output for the import() target!)
bugfix: inputs/GBIF/table.run: table.tsv.gz/make(): don't run table.tsv.gz/upload in test mode, to avoid clobbering the backup of a full table.tsv with a partial, testing table.tsv
lib/sh/db.sh: set test mode when using limited # rows
bugfix: inputs/GBIF/table.run: table.tsv.gz/upload(): don't use inplace mode because it leaves a newer mtime when aborted, causing rsync to think that the partial upload is actually newer than the source. note that rsync's --partial-dir mode is just as capable of resuming an aborted upload (it will just use a file in .rsync-tmp instead). inplace mode is primarily designed for fixed-offset files which don't change much between edits, but this is not true for exports (or the gzips of them), which will change the file offsets of most data if even one row or column is added or removed.
bugfix: inputs/GBIF/table.run: table.tsv.gz/make(): run table.tsv.gz/upload here instead of in table.tsv/make() because it should not run until table.tsv.gz is finished being made, which is not the case in table.tsv/make() because table.tsv.gz/make is run in the background
inputs/GBIF/table.run: table.tsv.gz/upload(): moved before table.tsv.gz/make() so it can be used by it
bugfix: inputs/GBIF/table.run: table.tsv.gz/upload(): need overwrite=1 because the mtime of an aborted inplace upload is newer
inputs/GBIF/table.run: table.tsv*/upload(): renamed to table.tsv.gz/upload() to upload only table.tsv.gz, not table.tsv, in order to save bandwidth
bugfix: lib/sh/sync.sh: also need to --include parent dirs for each --include path
lib/sh/util.sh: added path_parents()
*{.sh,run}: in comments, use ${array[@]} instead of @array for clarity
lib/sh/util.sh: foreach_arg(): moved `local a` to same line as for loop that uses it
bugfix: inputs/GBIF/table.run: table.tsv*/upload(): need to run put in live mode (live=1)
lib/sh/util.sh: foreach_arg(): echo_run the cmd at a log_level up so it isn't printed as if it were an external command (log_level 1)
lib/sh/sync.sh: removed `pf upload` debug statement
