Activity
From 05/16/2013 to 06/14/2013
06/14/2013
- 09:26 AM Task #636 (New): support TNRS-matching names in "", insted of encoding them and letting TNRS return no match
- * names like @"Acer rubrum"@ (_with_ quotes) currently get encoded as @ !quo Acer rubrum !quo @, which TNRS can't par...
- 07:42 AM Revision 9899: bugfix: /README.TXT: Full database import: added step to remove any leftover TNRS lockfile. usually, the PID in it would not exist, but sometimes it now refers to a different, active process which blocks tnrs.make.
06/13/2013
- 01:00 PM Revision 9898: schemas/vegbien.sql: allow public_ to view lookup tables (cultivated_family_locations, higher_plant_group_nodes)
- 08:13 AM Revision 9897: added backups/TNRS.backup.md5, vegbien.r9459.backup.md5
06/12/2013
- 02:45 PM Revision 9896: bugfix: lib/sh/local.sh: sync_upload(): need to use --no-group to prevent the group from being reset to aaronmk upon download from jupiter (which uses group aaronmk instead of bien). use ./fix_perms to set the group of all files to bien. also use --no-owner in case running as root.
- 02:12 PM Revision 9895: lib/sh/sync.sh: removed sync_download(). use swap=1 sync_upload() instead.
- 02:11 PM Revision 9894: lib/sh/sync.sh: removed download(). use swap=1 upload, or swap=1 upload_caller, instead. this avoids having separate upload()/download() pairs for every caller of upload(), because you can instead just set swap=1.
- 01:50 PM Revision 9893: bugfix: lib/sh/sync.sh: upload(): don't kw_params $swap because this unexports it, preventing put from seeing it. instead, use echo_vars to print it.
- 01:35 PM Revision 9892: added bin/sync_upload, a wrapper around sync_upload()
- 01:23 PM Revision 9891: added bin/sync_upload, a wrapper around sync_upload()
- 01:00 PM Revision 9890: bugfix: lib/sh/sync.sh: upload(): only add `--exclude="**"` if there are --includes. this enables running upload() without paths to upload all files.
- 12:56 PM Revision 9889: lib/sh/sync.sh: upload(): support passing -- options to put, which will not be run through the path->--include processing
- 12:55 PM Revision 9888: bugfix: lib/sh/sync.sh: upload(): added missing `local args=()` initializer
- 12:18 PM Revision 9887: /README.TXT: Full database import: On local machine: added step to do steps under Maintenance > "to synchronize vegbiendev, jupiter, and your local machine", which is needed in addition to `make inputs/upload` since that doesn't handle overwrites or deletions
- 12:10 PM Revision 9886: /README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: added warning that you should pay careful attention to all files that will be deleted or overwritten (as the three machines are often out of sync)
- 11:26 AM Revision 9885: added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.0.preamble.sql
- 11:17 AM Revision 9884: /README.TXT: Full database import: make inputs/{upload,download}: run them first with `test=1` to see what the changes will be
- 11:12 AM Revision 9883: /README.TXT: Full database import: `svn up`: use --force to avoid errors about existing files
- 10:49 AM Revision 9882: mappings/VegCore-VegBIEN.csv: genus->taxonlabel.taxonomicname: filter out genera that contain numbers (using new _filter_genus()), which break TNRS and prevent it from matching any other parts of the name. later, these genera can instead be moved to the end of the name, where TNRS will correctly match them as Unmatched_terms.
- 10:48 AM Revision 9881: bugfix: inputs/VegBIEN/: added _no_import to disable import for this folder, since this is actually just an entry in web/datasources/ with VegPath redirection links, rather than an input to the import process. this fixes "schema "VegBIEN" does not exist" errors generated in `make test`.
- 10:45 AM Revision 9880: inputs/input.Makefile: $(dontImport): also support putting a _no_import file at the top level in the datasource to exclude the entire datasource
- 10:30 AM Revision 9879: bugfix: lib/sh/local.sh: removed make() override, which is no longer needed now that its operations are performed by verbosity_compat(), and which caused errors by setting $verbosity to the invalid value ""
- 10:28 AM Revision 9878: bugfix: lib/sh/util.sh: verbosity_compat(): always use default verbosity (`unset verbosity`) when verbosity == 1, regardless of whether the caller has set $verbosity to the special value "", because $verbosity is supposed to be an integer field and "" is not supported by most functions that use $verbosity. in cases where a util.sh script is invoked, it will set $verbosity back to the default value 1, so this will function as before for util.sh scripts and fix $verbosity for scripts that use a different verbosity system.
- 10:05 AM Revision 9877: added inputs/GBIF/raw_occurrence_record_plants/table.tsv.md5
- 09:51 AM Revision 9876: inputs/GBIF/raw_occurrence_record_plants/test.xml.ref: regenerated. updated for new staging table input columns, which are now the same as the output columns.
- 09:41 AM Revision 9875: bugfix: inputs/input.Makefile: %/VegBIEN.csv: use header from map.csv instead of the new columns, so that source.shortname is set to GBIF instead of VegCore
- 09:24 AM Revision 9874: inputs/input.Makefile: %/VegBIEN.csv: when a runscript is available, instead map the *output* columns of map.csv to VegBIEN, because the columns have been renamed in the staging table
- 08:32 AM Revision 9873: inputs/GBIF/raw_occurrence_record_plants/VegBIEN.csv: regenerated, which adds row_num input col
- 08:16 AM Revision 9872: lib/sh/util.sh: echo_func(): check can_log at beginning of function, so that the resource-intensive func_loc (which calls `readlink -f`) does not need to be called if echo_cmd would not log anything at the current verbosity
- 08:02 AM Revision 9871: lib/sh/util.sh: echo_func(): removed no longer used $minor flag. use `clog++... echo_func` instead.
- 07:25 AM Revision 9870: lib/sh/util.sh: verbosity_compat(): don't make $verbosity a local var of this function, because then the changes will not be visible to the caller
- 07:12 AM Revision 9869: bugfix: bin/make: use verbosity_compat because some make-invoked commands (e.g. bin/map) don't support verbosity=""
- 07:10 AM Revision 9868: lib/sh/util.sh: command(): command__exec(): use verbosity_compat to support commands that don't support verbosity=""
- 07:09 AM Revision 9867: lib/sh/util.sh: added verbosity_compat(), for use with commands that don't support verbosity=""
- 06:47 AM Revision 9866: bugfix: lib/sh/local.sh: make(): when invoking overridden func, need make__make_sh
- 06:46 AM Revision 9865: bugfix: lib/sh/util.sh: self, self_sys aliases: need to remove any func_override suffix __* from the FUNCNAME
- 06:35 AM Revision 9864: bugfix: inputs/GBIF/import_order.txt, run: updated raw_occurrence_record/ to raw_occurrence_record_plants/
- 06:32 AM Revision 9863: inputs/FIA/occurrence_all/test.xml.ref: update inserted row count
- 06:27 AM Revision 9862: bugfix: bin/make: include local.sh so that its default verbosity-setting make() override will be used
- 06:26 AM Revision 9861: lib/sh/local.sh: added make() override, which uses the default verbosity (i.e. verbosity="") when verbosity == 1. scripts that use lib/sql.py (which uses $verbosity) have different default verbosities, and this default should not be overriden by an env var, unless a higher verbosity has been set.
- 06:24 AM Revision 9860: lib/sh/local.sh: added missing include of make.sh, used by root_make()
- 05:57 AM Revision 9859: schemas/vegbien.sql: added _filter_genus()
- 04:47 AM Revision 9858: inputs/GBIF/raw_occurrence_record_plants/run: import() runtime: specified that this does not include table.tsv.gz/make()
- 04:07 AM Revision 9857: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: # duplicates: added revision #
- 04:07 AM Revision 9856: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: documented that there are 4.5 million duplicates (59,998,354 rows before - 55,417,646 rows after = 4,580,708)
- 03:49 AM Revision 9855: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: added rerun time (~0 thanks to index, so no problem doing the DELETE each time postprocess.sql is run)
- 03:25 AM Revision 9854: *{.sh,run}: use simpler .rel() instead of `. "$(dirname "${BASH_SOURCE[0]}")"/...` for relative includes
- 03:24 AM Revision 9853: lib/sh/util.sh: added .rel()
- 02:52 AM Revision 9852: schemas/VegCore/VegCore.pg.sql: regenerated from installed schema. a Linux bug that caused constraints to be output in reverse order has now been fixed, as of 12.04.2 (broken in 12.04.1).
- 02:48 AM Revision 9851: bugfix: inputs/GBIF/_MySQL/MySQL_schema, MySQL_data: sed: put {} commands on their own line to work on Mac
- 02:11 AM Revision 9850: lib/sh/util.sh: .(): put echo_func on its own line for clarity
- 02:10 AM Revision 9849: lib/sh/util.sh: .(): added echo_func (2 log_levels up because it's internal)
- 02:06 AM Revision 9848: schemas/VegCore/mk_derived: use new lib/sh/local.sh instead of lib/import.sh (a precursor to util.sh, etc. still used by inputs/FIA/)
06/11/2013
- 07:50 PM Revision 9847: bugfix: load_data(): verbosity_min: use verbosity_min='' so that csv2db's default verbosity (3) is used, instead of setting the verbosity directly to 3, which caused the log++ logging output from bin/make to be echoed at verbosity 3, creating cluttered output
- 07:43 PM Revision 9846: lib/sh/util.sh: verbosity_min(): support value '', which sets verbosity=''
- 06:40 PM Revision 9845: bugfix: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: updated column names to match the renamings in map.csv, which are now performed on the staging table itself
- 06:38 PM Revision 9844: lib/sh/util.sh: run_args_cmd(): time the command so that the runtime of the outer runscript target (i.e. the command run from the shell) is printed at the end of the output, like in bin/make
- 05:56 PM Revision 9843: bugfix: inputs/input.Makefile: %/install: don't run $(cleanup) if it has already been run by $(import_install_), so that it doesn't run twice
- 05:54 PM Revision 9842: inputs/input.Makefile: %/postprocess: don't run postprocess.sql if it is supposed to be run by a runscript, because postprocess.sql may then depend on additional steps the runscript runs before it
- 05:25 PM Revision 9841: lib/runscripts/table.run: import(): use self_make on load_data so that the remake status determines whether the table is reinstalled
- 05:23 PM Revision 9840: bugfix: lib/runscripts/mysql.table.run: import(): added missing set_make_vars, needed by self_make
- 05:21 PM Revision 9839: bugfix: lib/runscripts/table.run: load_data(): need to use $_remake instead of $remake when using set_make_vars
- 05:20 PM Revision 9838: lib/runscripts/table.run: added set_make_vars to all make targets so $remake would be propagated appropriately
- 05:18 PM Revision 9837: lib/runscripts/table.run: load_data(): also clobber install log if remaking, because the table will be reinstalled
- 05:12 PM Revision 9836: lib/runscripts/table.run: load_data(): automatically select noclobber mode depending on whether the install log already exists. this removes the need for a separate load_data_first_run() function.
- 04:59 PM Revision 9835: bugfix: lib/runscripts/table.run: load_data(): ignore errors if table already exists
- 04:52 PM Revision 9834: lib/runscripts/table.run: load_data(): use noclobber=1 to avoid overwriting the install log when re-running the install target idempotently. load_data_first_run() is now available to preserve the output in the log on the first run.
- 04:51 PM Revision 9833: inputs/input.Makefile: Staging tables installation: $(logInstall): don't output to the install log if $noclobber flag is set, to prevent overwriting the log when re-running the install target idempotently
- 04:18 PM Revision 9832: bugfix: lib/runscripts/mysql.table.run: import(): move previous versions of table.tsv out of the main dir before loading staging tables, to prevent them from being considered a TSV segment file and prepended to table.tsv
- 04:15 PM Revision 9831: lib/sh/util.sh: added mv2dir(), mv_glob which wrap mv
- 04:14 PM Revision 9830: lib/sh/util.sh: added mkdir alias which adds -p to prevent errors if the dir already exists
- 03:45 PM Revision 9829: lib/sh/util.sh: added wildcard alias, similar to make's $(wildcard) function
- 03:29 PM Revision 9828: bugfix: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: institution_code index: create it idempotently using create_if_not_exists() and an explicit index name, so that a duplicate index doesn't get added each time postprocess.sql is run
- 03:27 PM Revision 9827: lib/sh/local.sh: psql(): don't put util in the search_path because psql scripts now add it themselves if they need it, using `SELECT util.search_path_append(util);`
- 03:22 PM Revision 9826: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: add util to the search_path so that postprocess.sql will also work when run by inputs/input.Makefile, which only puts the datasource (GBIF) in the search_path
- 03:18 PM Revision 9825: schemas/util.sql: added search_path_append()
- 03:12 PM Revision 9824: schemas/util.sql: added eval() to allow running EXECUTE outside of a function (and to echo the command that is run)
- 09:04 AM Revision 9823: inputs/GBIF/raw_occurrence_record_plants/run: added import() runtime (5 h)
06/10/2013
- 11:58 PM Revision 9822: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv.gz/make() runtime: noted that this excludes the upload time
- 11:58 PM Revision 9821: inputs/GBIF/raw_occurrence_record_plants/run: added table.tsv.gz/upload() runtime (15 min)
- 11:48 PM Revision 9820: added lib/runscripts/mysql.table.run (general to all MySQL datasources) and use it in inputs/GBIF/table.run
- 11:13 PM Revision 9819: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): to view runtime when using `screen`: keys used to scroll: added Ctrl-B/Ctrl-F for page-at-a-time scrolling (there are a lot of pages of output for the import() target!)
06/09/2013
- 09:21 PM Revision 9818: bugfix: inputs/GBIF/table.run: table.tsv.gz/make(): don't run table.tsv.gz/upload in test mode, to avoid clobbering the backup of a full table.tsv with a partial, testing table.tsv
- 09:18 PM Revision 9817: lib/sh/db.sh: set test mode when using limited # rows
- 09:08 PM Revision 9816: bugfix: inputs/GBIF/table.run: table.tsv.gz/upload(): don't use inplace mode because it leaves a newer mtime when aborted, causing rsync to think that the partial upload is actually newer than the source. note that rsync's --partial-dir mode is just as capable of resuming an aborted upload (it will just use a file in .rsync-tmp instead). inplace mode is primarily designed for fixed-offset files which don't change much between edits, but this is not true for exports (or the gzips of them), which will change the file offsets of most data if even one row or column is added or removed.
- 09:01 PM Revision 9815: bugfix: inputs/GBIF/table.run: table.tsv.gz/make(): run table.tsv.gz/upload here instead of in table.tsv/make() because it should not run until table.tsv.gz is finished being made, which is not the case in table.tsv/make() because table.tsv.gz/make is run in the background
- 08:59 PM Revision 9814: inputs/GBIF/table.run: table.tsv.gz/upload(): moved before table.tsv.gz/make() so it can be used by it
- 08:39 PM Revision 9813: bugfix: inputs/GBIF/table.run: table.tsv.gz/upload(): need overwrite=1 because the mtime of an aborted inplace upload is newer
- 08:32 PM Revision 9812: inputs/GBIF/table.run: table.tsv*/upload(): renamed to table.tsv.gz/upload() to upload only table.tsv.gz, not table.tsv, in order to save bandwidth
- 08:28 PM Revision 9811: bugfix: lib/sh/sync.sh: also need to --include parent dirs for each --include path
- 08:27 PM Revision 9810: lib/sh/util.sh: added path_parents()
- 08:01 PM Revision 9809: *{.sh,run}: in comments, use ${array[@]} instead of @array for clarity
- 07:54 PM Revision 9808: lib/sh/util.sh: foreach_arg(): moved `local a` to same line as for loop that uses it
- 07:00 PM Revision 9807: bugfix: inputs/GBIF/table.run: table.tsv*/upload(): need to run put in live mode (live=1)
- 06:53 PM Revision 9806: lib/sh/util.sh: foreach_arg(): echo_run the cmd at a log_level up so it isn't printed as if it were an external command (log_level 1)
- 06:33 PM Revision 9805: lib/sh/sync.sh: removed `pf upload` debug statement
- 06:32 PM Revision 9804: bugfix: lib/sh/util.sh: set_fds(): localize $i so it doesn't overwrite any previous value
- 06:30 PM Revision 9803: inputs/GBIF/table.run: table.tsv/make(): run table.tsv*/upload when the file make is done so that the file is backed up to jupiter
- 06:29 PM Revision 9802: inputs/GBIF/table.run: added table.tsv*/upload()
- 06:27 PM Revision 9801: lib/sh/local.sh: added sync_upload(), sync_download() with $sync_local_dir, $sync_remote_url config vars
- 06:26 PM Revision 9800: added lib/sh/sync.sh with upload(), download()
- 06:25 PM Revision 9799: lib/sh/util.sh: added foreach_arg()
- 06:06 PM Revision 9798: bugfix: lib/sh/util.sh: need to use `declare -p` instead of ${var+isset} because ${var+isset} returns not set for empty arrays
- 06:05 PM Revision 9797: lib/sh/util.sh: added echo_vars() stub
- 05:43 PM Revision 9796: lib/sh/util.sh: added echo_run() stub
- 05:39 PM Revision 9795: lib/sh/util.sh: set_fds(): don't run (or echo) exec if no redirections are being made
- 05:37 PM Revision 9794: bugfix: lib/sh/util.sh: added missing stub for indent alias (used by echo_func alias, which is a stub). without the stub, /usr/bin/indent would be used instead on Mac.
- 04:01 PM Revision 9793: bugfix: lib/sh/local.sh: root_rel_path(): added echo_func
- 04:01 PM Revision 9792: bugfix: lib/sh/local.sh: root_rel_path(): use canon_rel_path instead of rel_path because $1 may be absolute rather than relative to the current dir, so $root_dir needs to be absoluted (which requires $1 to be absoluted as well)
- 03:56 PM Revision 9791: lib/sh/util.sh: support custom $base_dir, which will be run through realpath() to match $path ($PWD, which was used before, did not need to be realpath'd because it was already absolute)
- 03:34 PM Revision 9790: lib/sh/util.sh: moved echo_func alias to stub because it must be embedded in its expanded alias form to work properly
- 03:29 PM Revision 9789: lib/sh/util.sh: declare echo_func as a stub before it's defined, so that functions can use it even if they are defined before it (and its logging functionality will be enabled as soon as it's defined)
- 03:26 PM Revision 9788: lib/sh/util.sh: rel_path(): don't log++ it, and instead only log++ applicable calls of it or its callers. this allows non-internal calls of rel_path() to be logged at the usual log_level.
- 01:00 PM Revision 9787: lib/sh/local.sh: added root_rel_path()
- 12:39 PM Revision 9786: lib/Firefox_bookmarks.reformat.csv: unescape HTML in page's description, such as links to more info. this is necessary to properly render the persistent shells link in the screen > scrollback folder description.
- 12:37 PM Revision 9785: bin/repl: added unescape_html() filter function, which can be specified as the replacement string
- 12:35 PM Revision 9784: bin/repl: support Unicode characters in the matched portion of the string
- 11:52 AM Revision 9783: web/links/index.htm: updated to Firefox bookmarks. `screen`: scrollback: added link to our wiki page on persistent shells, which are a better way to support reconnecting.
- 11:36 AM Revision 9782: web/links/index.htm: updated to Firefox bookmarks. added bookmarks about the `screen` command, especially how to access the scrollback. resorted several folders alphabetically.
- 11:13 AM Revision 9781: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): documented how to view the runtime when using `screen` (press Ctrl-A [ , use up-arrow, and then press Esc to leave copy mode)
- 11:12 AM Revision 9780: inputs/GBIF/raw_occurrence_record_plants/run: herbaria_filter/make(): use new ih_herbarium table instead of the herbaria_filter.ih.csv_ file directly
06/08/2013
- 12:23 PM Revision 9779: inputs/GBIF/raw_occurrence_record_plants/run: added ih_herbarium/make(), which stores the IH herbaria
- 11:50 AM Revision 9778: bugfix: inputs/GBIF/raw_occurrence_record_plants/run: table/make(): also filter out rows with a non-plant family (as described at http://vegpath.org/wiki/2013-06-06_conference_call#GBIF-subsetting-fix-raw_occurrence_record-filter-formula), since some institutions have both animal and plant rows, even though they are in IH or in the 80% list. (note that NULL families are OK.)
- 04:12 AM Revision 9777: *{.sh,run}: use mysql instead of mysql_ANSI because mysql is now an alias to mysql_ANSI (since ANSI mode still supports key MySQL features, like `` quotes)
- 04:09 AM Revision 9776: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): documented that incremental output is provided right away with --quick (unbuffered), but takes awhile to become visible in Macfusion sshfs. this can be tested with `while true; do stat inputs/GBIF/raw_occurrence_record_plants/table.tsv; sleep 2; done` running concurrently with `./inputs/GBIF/raw_occurrence_record_plants/run table.tsv/make` on vegbiendev:/home/bien/svn .
- 04:00 AM Revision 9775: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): use new raw_occurrence_record_plants view from table/make()
- 03:15 AM Revision 9774: bugfix: inputs/GBIF/raw_occurrence_record_plants/run: table/make(): added make of prerequisites
- 03:14 AM Revision 9773: bugfix: inputs/GBIF/raw_occurrence_record_plants/run: table/make(): don't reset $table to plant_fraction_for_herbaria_filter for commands that use $table
- 03:10 AM Revision 9772: inputs/GBIF/raw_occurrence_record_plants/run: added table/make(), which makes the filter view
- 02:14 AM Revision 9771: inputs/GBIF/raw_occurrence_record/: renamed to raw_occurrence_record_plants because it's actually only the plants in raw_occurrence_record, not all of raw_occurrence_record. also, this will allow us to create a separate raw_occurrence_record_plants view whose name matches the folder and does not collide with the raw_occurrence_record table.
- 12:44 AM Revision 9770: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): added runtime, which is ~0 since it just needs to do CSV import and index scans
- 12:43 AM Revision 9769: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): time the population of herbaria_filter
06/07/2013
06/06/2013
- 04:54 PM Revision 9767: inputs/.TNRS/schema.sql: tnrs_populate_fields(): documented runtime (17 min)
- 04:49 PM Revision 9766: bugfix: inputs/GBIF/raw_occurrence_record/run: plant_fraction/make(): plant_fraction column: COUNT(boolean) counts *non-NULL* rather than true values (which counter-intuitively includes false, because it's non-NULL), so need to add NULLIF(..., false) around the boolean expression to turn it into a NULL-or-not expression. see http://vegpath.org/wiki/2013-06-06_conference_call#GBIF-subsetting-fix-plant_fraction-SQL-bug .
- 04:29 PM Revision 9765: inputs/.TNRS/schema.sql: tnrs_populate_fields(): documented that when changing this function, you must regenerate the derived cols using `UPDATE tnrs SET "Name_submitted" = "Name_submitted"`
- 04:21 PM Revision 9764: inputs/.TNRS/schema.sql: tnrs_populate_fields(): Is_plant: must match family as Family_score = 1 (as discussed during conference call vegpath.org/wiki/2013-05-30_conference_call#postprocess-TNRS-results-to-exclude-animals-with-genus-homonyms) instead of as Family_matched IS NOT NULL (as listed in Brad's formula at vegpath.org/wiki/Result_filtering#TNRS-results) because TNRS transforms animal to plant families via fuzzy matching, necessitating a Family_score check to ensure an exact match to a plant family that was not transformed from an animal family
- 03:29 PM Revision 9763: inputs/.TNRS/schema.sql: added Is_plant derived field, which is populated using the formula at vegpath.org/wiki/Result_filtering#TNRS-results . note that the homonym filtering is currently excluded until we determine whether we can get direct access to the IRMNG homonyms database (http://www.cmar.csiro.au/datacentre/irmng/homonyms.htm). note also that changes to the TNRS schema cannot be fully tested until any TNRS client bugs are fixed, because the data.sql updater requires a working TNRS client to regenerate the sample data.
- 02:42 PM Revision 9762: inputs/.TNRS/schema.sql: updated for current TSV schema: renamed Accepted_species->Accepted_name_species, Accepted_family->Accepted_name_family
- 02:27 PM Revision 9761: bugfix: schemas/vegbien.sql: tnrs_input_name: must anti-join against MatchedTaxon rather than ValidMatchedTaxon to ensure that *all* of TNRS.tnrs is excluded from the input names. this prevents duplicates from appearing in the TNRS results, which would break the TSV import into TNRS.tnrs. it also prevents no-match names from being scrubbed repeatedly because they were not properly filtered out of the input names.
- 02:17 PM Revision 9760: inputs/.TNRS/schema.sql: fixed whitespace
- 02:15 PM Revision 9759: inputs/.TNRS/schema.sql: added MatchedTaxon view, which now just renames the columns but does not filter the results, and use it in ValidMatchedTaxon
- 02:11 PM Revision 9758: inputs/.TNRS/schema.sql: MatchedTaxon: renamed to ValidMatchedTaxon since this view actually contains only the names with a valid match
- 12:59 PM Revision 9757: bugfix: lib/sql.py: parse_exception(): make_DuplicateKeyException(): handle nested exceptions (which should never be generated, but may be in case of sql.py bugs such as wiki.vegpath.org/To_Do#Fixes > #1) by printing the nested exception and then rethrowing the original exception, so that the original exception does not get lost and still ends up at the end of the program's output, to enable debugging
- 11:08 AM Revision 9756: inputs/.TNRS/schema.sql: tnrs: documented that when changing this table's schema, you must regenerate data.sql using `inputs/test_taxonomic_names/test_scrub`
- 08:09 AM Revision 9755: inputs/GBIF/raw_occurrence_record/run: table.tsv.gz/make(): documented runtime (35 min)
- 07:52 AM Revision 9754: bugfix: schemas/vegbien.sql: analytical_stem_view: speciesBinomialWithMorphospecies: if accepted name not specified, use matched name (matched*) or Name_submitted (concatenatedScientificName), as described at http://wiki.vegpath.org/2013-05-30_conference_call#fix-TNRS-speciesBinomialWithMorphospecies-to-include-alternatives-when-no-accepted-name
- 07:24 AM Revision 9753: lib/sh/make.sh: make(): time all invocations of make
- 07:14 AM Revision 9752: lib/sh/make.sh: make() at verbosity < 4, hide messages about making included Makefiles: use sed with a range expression (/.../,/.../) to also exclude all log messages between an opening "make ...Makefile" and a closing "make[#]: ...Makefile"
- 06:35 AM Revision 9751: lib/sh/util.sh: log+ aliases: added clog++/-- aliases for cmds, which don't include log_local. these are useful when you can't just use "log++" because you need the command following it to be alias-expanded.
- 05:52 AM Revision 9750: bugfix: lib/sh/make.sh: make(): use [:char_class:] exprs instead of \X char class abbrs because the \ abbrs are not supported on Linux
- 05:33 AM Revision 9749: inputs/GBIF/table.run: table.tsv/make(): remake table.tsv.gz/make() after table.tsv is made
- 05:27 AM Revision 9748: inputs/GBIF/table.run: added table.tsv.gz/make()
- 05:26 AM Revision 9747: lib/sh/util.sh: added bg_cmd() and helpers log_bg(), log_last_bg()
- 05:04 AM Revision 9746: lib/runscripts/subdir.run: auto-append -remake to all targets in remake mode
- 04:15 AM Revision 9745: lib/runscripts/subdir.run: subdir_make(): put prepending of "$subdir/" on its own line for clarity
- 04:00 AM Revision 9744: bugfix: lib/common.Makefile: gzip/gunzip: in `touch -r`, *$<* needs to be the reference file
- 02:36 AM Revision 9743: *{.sh,run}: make() calls: removed no longer applicable silent=1, which is now handled automatically by the log_level mechanism
- 02:35 AM Revision 9742: /Makefile: new config target (part of install): install our bin/make in the user's ~/bin dir so that bin/make (with filtering of verbose messages) will always be used instead of standard make (without us needing to change every occurrence of make to bin/make!)
- 02:25 AM Revision 9741: bugfix: bin/make: use standard make logging port (1) instead of $log_fd (30) so that the output of bin/make can in turn be filtered by util.sh using the standard cmd_log_fd=1
- 02:23 AM Revision 9740: lib/sh/util.sh: command(): print the log_fd in effect in case the user changes it
- 02:23 AM Revision 9739: lib/sh/make.sh: make(): log the messages about making included Makefiles at log_level 4 instead of 3, so that verbosity 3 can be used to see the vars being used by the filtering process (i.e. the filtering process must still be enabled at verbosity 3)
- 02:00 AM Revision 9738: bin/make: use `readlink -f` on BASH_SOURCE[0] so that this script can also be run via a symlink
- 01:57 AM Revision 9737: lib/sh/util.sh: no_PATH_recursion(): also remove $top_symlink_dir_abs
- 01:56 AM Revision 9736: lib/sh/util.sh: added $top_symlink_dir_abs
- 01:53 AM Revision 9735: lib/sh/util.sh: $top_dir_orig: renamed to $top_symlink_dir since this reflects better what the dir actually represents, and when it differs from $top_dir
- 01:33 AM Revision 9734: lib/sh/make.sh: make(): fixed func name in usage
- 01:32 AM Revision 9733: bin/make: don't print make cmd by default, so that only `make` output is printed at verbosity 1
- 01:31 AM Revision 9732: bin/make: don't reinvoke make() if the make filter has already been set up, as indicated by $is_outermost (instead, invoke make directly using exec)
- 01:30 AM Revision 9731: lib/sh/util.sh: added $is_outermost, which stores whether the util.sh env has not yet set up
- 01:28 AM Revision 9730: lib/sh/util.sh: echo_redirs_cmd(): added $cmd_name_log_inc kw param to allow echoing the command name at a different log_level than the command output
- 12:22 AM Revision 9729: lib/sh/make.sh: make(): log the filter setup operations at log_level 2 so they do not clutter up normal output
- 12:13 AM Revision 9728: lib/sh/util.sh: filter_fd: alias-expand filter_cmd
- 12:08 AM Revision 9727: added bin/make, which runs make, hiding verbose messages about making included Makefiles. this should be used in preference to plain make, to avoid excessive log messages that prevent the user from seeing the core commands that are being run.
- 12:05 AM Revision 9726: lib/sh/util.sh: added no_PATH_recursion(), which allows running a system command of the same name as the script
- 12:04 AM Revision 9725: lib/sh/util.sh: added PATH_rm(), which removes components from the PATH
- 12:03 AM Revision 9724: lib/sh/util.sh: added $top_dir_abs, $top_dir_orig
06/05/2013
- 11:44 PM Revision 9723: bugfix: lib/sh/util.sh: command(): use `builtin command` instead of `exec` so that options like -p (reset PATH) work properly. also, the command builtin it overrides is designed to be used with more than just external commands, and command() should not impose additional limitations.
- 10:54 PM Revision 9722: lib/sh/util.sh: added self_sys alias, which uses only system utilities (`command -p`) instead of the current PATH
- 10:37 PM Revision 9721: lib/sh/make.sh: make(): at verbosity < 3, hide messages about making included Makefiles. this makes the make output **much** more readable when a Makefile contains an include statement, because there won't be a ton of log messages every time a Makefile is included. this filtering is so useful that it probably makes sense to run make for any of our Makefiles using `lib/runscripts/util.run make ...` instead of plain make. compare e.g. `make inputs/ACAD/Specimen/map.csv` (53 lines of output) and `lib/runscripts/util.run make inputs/ACAD/Specimen/map.csv` (17 lines of output, 1/3 as much).
- 10:15 PM Revision 9720: lib/sh/util.sh: log++: before cmd: documented that you need to use "log++" instad of log++ to avoid using the log++ alias, which prepends a log_local call. omitting the quotes is generally not a problem, but when there is another command wrapping the log++, you need the "" to avoid the wrapper applying to log_local's declare call instead.
- 10:09 PM Revision 9719: lib/sh/util.sh: log++: before cmd: documented how to use it as `log+ #` when incrementing multiple log_levels at once (this is a better method than `"log++" "log++" ...`)
- 10:04 PM Revision 9718: lib/sh/util.sh: log++: before cmd: documented that you need to use it as `"log++" "log++"` when repeating it multiple times
- 10:02 PM Revision 9717: lib/sh/util.sh: added filter_fd(), which encapsulates the use of >() process substitution for filtering an fd other than stdout (yes, this is possible without lots of 3>&1 1>&2 2>&3 redirections!). this can be useful e.g. to filter logging output or highlight errors.
- 09:49 PM Revision 9716: lib/sh/util.sh: log+(): documented that with a cmd, assignments are applied just to the cmd, so log_local is not needed
- 08:59 PM Revision 9715: lib/sh/util.sh: moved pipe_delay() before fd-related functions so it can be used by them
- 08:56 PM Revision 9714: lib/sh/util.sh: removed no longer needed load new aliases before echo_stdin(), since pipe_delay() is now a function
- 08:48 PM Revision 9713: bugfix: lib/sh/util.sh: set_fds(): add #<>&- before every #<>&# reopen: need to use loop var $i instead of $1 (which would have been used with a while/shift method of iterating over $@)
- 12:57 PM Revision 9712: bugfix: lib/sh/util.sh: set_fds(): added workaround for strange bash bug where reopening an fd sometimes first requires explicitly closing it, by adding an <>&- entry for every redirection
- 12:53 PM Revision 9711: lib/sh/util.sh: added match_prefix()
- 03:28 AM Revision 9710: lib/sh/make.sh: make(): determine --silent status based on the verbosity (<=0) instead of a kw param
- 03:11 AM Revision 9709: inputs/GBIF/table.run: table.tsv.md5/make(): only use %-remake if remaking
- 03:10 AM Revision 9708: lib/sh/make.sh: make(): removed extra space after --silent
06/04/2013
- 11:17 PM Revision 9707: lib/sh/db.sh: moved mysql_root() after the mysql->mysql_ANSI alias (and load new aliases) so that it will also use ANSI mode and support "" identifiers
- 11:15 PM Revision 9706: lib/sh/db.sh: mysql: always use ANSI mode, to support "" identifiers. note that `` are *still* supported in this mode, so it also works with SHOW CREATE TABLE output and dumpfiles.
- 11:05 PM Revision 9705: bugfix: inputs/GBIF/raw_occurrence_record/run: plant_fraction_for_herbaria_filter/make(): need to make prerequisites first (plant_fraction/make)
- 06:35 PM Revision 9704: bugfix: inputs/GBIF/table.run: table.tsv.md5/make(): use %-remake to ensure that the .md5 file is remade, regardless of the .md5 file's mtime relative to table.tsv. you would generally expect table.tsv's new mtime to be newer than the .md5 file's (thus triggering make to run), but if you e.g. ran svn up after making the table.tsv, this might not be the case.
- 06:31 PM Revision 9703: /Makefile: moved %/remake, %-remake to lib/common.Makefile because they are generally useful
- 06:26 PM Revision 9702: /Makefile: moved %/reinstall to lib/common.Makefile because it is generally useful
- 06:07 PM Revision 9701: lib/sh/db.sh: mysql_cmd(): run the command with `time` because in mysql()'s output_data mode, no queries, and therefore no runtimes, are echoed, so the total runtime needs to be echoed separately instead. the total runtime is also useful in general, when many long-running queries are run and you would also like to know the total time (e.g. in make_analytical_db).
- 05:54 PM Revision 9700: lib/sh/db.sh: limit(): usage: surrounded query in "" to clarify that it's a string, not a command
06/01/2013
- 09:32 PM Revision 9699: inputs/GBIF/table.run: table.tsv/make(): use new set_large_table to prevent table.tsv from being deleted on error for full export runs (while still deleting it on error for sample subset runs)
- 09:31 PM Revision 9698: lib/sh/db.sh: added set_large_table alias, used to set to_file's $del flag based on $limit
- 09:30 PM Revision 9697: lib/sh/util.sh: added exit2bool()
- 09:18 PM Revision 9696: lib/sh/util.sh: int2bool(): renamed to int2exit() because it actually sets a boolean exit status rather than returning a boolean string
- 09:07 PM Revision 9695: bugfix: lib/sh/util.sh: to_file(): also kw_params the other kw params if_not_exists, del
- 07:02 PM Revision 9694: lib/sh/db.sh: mysql_cmd(): use --quick to avoid buffering entire result (which is slow and memory-intensive for large result sets). this option applies to both mysql() and mysqldump().
- 06:35 PM Revision 9693: lib/sh/make.sh: self_make(): documented that it should be preceded by set_make_vars
- 06:34 PM Revision 9692: bugfix: inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.run: ^.preamble.sql/make(): need to run set_make_vars even though the make vars are not used, because set_make_vars sets $_remake, which is needed by self_make
- 06:33 PM Revision 9691: lib/sh/make.sh: set_make_vars: echo_vars $_remake to help debugging
- 06:27 PM Revision 9690: bugfix: lib/sh/util.sh: to_file(): need to run invoked cmd using redir so that >$stdout redir is applied properly when cmd is a shell function instead of an external command (external commands were already redirected properly because command() calls redir). this fixes a bug in inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.run ^.preamble.sql/make(), where the generated file would be output to stdout instead of to the file because to_file()'s command was a shell function, and therefore the redirection was not applied. this fix requires redir() to be a separate function from command(), because command() does many things that are only applicable to external commands.
- 06:20 PM Revision 9689: lib/sh/util.sh: redir(): override save_e and add `unset redirs` so error handlers are not redirected
- 06:15 PM Revision 9688: lib/sh/util.sh: added alias_append(), similar to func_override() for aliases
- 06:05 PM Revision 9687: lib/sh/util.sh: redir(): don't do redir actions if redir will be run later (i.e. if command to run starts with `redir` or `command`)
- 06:04 PM Revision 9686: lib/sh/util.sh: redir(), command(): log their function calls at the usual log_level (2) instead of one higher, so that they are printed in the call tree to help debugging
- 05:39 PM Revision 9685: lib/sh/util.sh: added redir() and use it in command() to perform and echo the redirections
- 09:35 AM Revision 9684: lib/sh/util.sh: command(): removed comment that "the following redirections must happen in exactly this order", because there is now only one redirection
- 06:01 AM Revision 9683: lib/sh/util.sh: command(): removed 2>&$err_fd, which is no longer needed because $err_fd is now 2 (in general, there is probably not a need for a special $err_fd var, because 2 is already stderr)
- 05:52 AM Revision 9682: lib/sh/util.sh: command(): don't set err_fd to global stderr, because this prevents errors from being captured and parsed by callers. it is also not necessary to use a separate port than stderr, because stderr already contains only errors (since logging messages go to their own port). global stderr would still be useful e.g. for displaying input prompts the user when reading from global stdin.
- 05:17 AM Revision 9681: bugfix: *run: overriding targets: use new self_make to properly progagate the $remake flag to the overridden target, so that the target itself is not skipped
- 05:15 AM Revision 9680: lib/sh/util.sh: to_file(): added del= flag to prevent the file from being auto-removed on error (e.g. to preserve a partial result, which normally would be removed)
- 05:12 AM Revision 9679: lib/sh/make.sh: added self_make(), which progagates the $remake flag (normally it is not progagated, because prerequisites should not also be remade)
- 03:58 AM Revision 9678: lib/sh/util.sh: usage comments: when there is a descriptive comment on the same line as the usage, prepend it with # (as if it were an end-of-line comment) instead of enclosing it in (), to make it visually obvious that it's a comment and not part of the usage commands
- 03:54 AM Revision 9677: lib/sh/util.sh: added all_funcs(), which lists all declared functions
- 03:31 AM Revision 9676: bugfix: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added check_target_exists so table.tsv would not be overwritten if it already existed
- 03:30 AM Revision 9675: bugfix: inputs/GBIF/table.run: table.tsv/make(): force table.tsv.md5 to be remade (using remake=1) because the table.tsv contents will have changed
- 03:29 AM Revision 9674: bugfix: lib/sh/make.sh: set_make_vars: can't use end-of-line comment in alias because it will comment out whatever is after the alias where it's used. can't just put a newline or ; after the end-of-line comment because the alias's lines will be combined onto one line using ; , so end-of-line comments would not be supported.
- 03:23 AM Revision 9673: lib/sh/util.sh: added pf(), to print a function declaration for debugging
- 02:08 AM Revision 9672: inputs/GBIF/raw_occurrence_record/run: plant_fraction/make(): documented runtime (1 hr)
- 01:19 AM Revision 9671: *.url: mailto URLs: use the standard e-mail dotpath syntax e-mail@host?name.date.subject.(attachment) (vegpath.org/wiki/Global_IDs#Resource). populated missing fields (e.g. name, subject) where needed.
- 12:12 AM Revision 9670: *.url: mailto URLs: ensured that they are proper URLs with the mailto: protocol
05/30/2013
- 08:32 PM Revision 9669: lib/sh/util.sh: to_file(): exc handler that rm's file: unset redirs so it isn't used in the rm cmd
- 08:22 PM Revision 9668: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): removed no longer needed explicit clear of $remake, which is now done by make.sh instead
- 08:18 PM Revision 9667: lib/sh/make.sh: set_make_vars: don't progagate remake to prerequisites, so that remake=1 only applies to the outermost target rather than forcing every prerequisite to be remade, too
- 07:58 PM Revision 9666: lib/sh/make.sh: moved remaking section before set_make_vars so that it can be used in set_make_vars
- 07:53 PM Revision 9665: inputs/GBIF/raw_occurrence_record/run: added herbaria_filter/seal()
- 07:51 PM Revision 9664: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): changed "from IH" to "contains all of IH" because not all rows are now from IH
- 07:49 PM Revision 9663: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): renamed acronym->institution_code to match the column name in raw_occurrence_record rather than in IH
- 07:46 PM Revision 9662: inputs/GBIF/raw_occurrence_record/run: removed no longer used herbaria_filter.plant_fraction.csv_/make(). use plant_fraction_for_herbaria_filter view instead.
- 07:45 PM Revision 9661: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): use the plant_fraction_for_herbaria_filter view directly instead of first exporting it to a CSV
- 07:19 PM Revision 9660: lib/sh/db.sh: mysql_import(): in append mode, use LOAD DATA IGNORE to allow inserting duplicate rows
- 07:09 PM Revision 9659: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): if remaking, turn off remake mode after doing this target's rm operations, so that prerequisite targets are not also remade
- 06:56 PM Revision 9658: lib/sh/util.sh: to_file(): removed no longer needed separate logging of >$stdout, which is now done by command()
- 06:50 PM Revision 9657: lib/sh/util.sh: echo_redirs_cmd(): use $@ in a subshell instead of manipulating the @redirs array directly, because operations on $@ (e.g. $#, $1, shift) are much simpler than the corresponding array operations ( ${#redirs[@]}, ${redirs[0]}, redirs=("${redirs[@]:1}") )
- 06:42 PM Revision 9656: lib/sh/util.sh: echo_redirs_cmd(): log each file redir with a separate log() statement, so each line is indented
- 06:38 PM Revision 9655: lib/sh/util.sh: added echo_redirs_cmd and use it in command() to print cmd
- 06:32 PM Revision 9654: lib/sh/util.sh: command(): print <>file redirects before command, because they introduce it
- 06:31 PM Revision 9653: lib/sh/util.sh: added starts_with()
- 05:49 PM Revision 9652: lib/sh/util.sh: to_file(): use @redirs to echo and set >$stdout instead of setting it manually, which is possible now that the command() @redirs bug has been fixed
- 05:43 PM Revision 9651: bugfix: lib/sh/util.sh: convention of fds to use for command-specific alternate stdin/stdout/stderr: changed to 40/41/42 because 10/11/12 are used by eval (which is used by set_fds()). use of fd 10/11/12 will cause hard-to-find silent bugs because exec will not print an error when these are used. documented why not to use other series of fds for this purpose:
- # do NOT use 1x, which are used by eval (which is used by set_fds())
# do NOT use 2x, which are used as global stdin/... - 02:43 PM Revision 9650: lib/sh/local.sh: psql(): use new psql() from db.sh instead of psql_script_vegbien/psql_verbose_vegbien. this requires setting local_pg_database=vegbien to replace vegbien_dest used by psql_*_vegbien.
- 02:38 PM Revision 9649: lib/sh/db.sh: psql(): set $PG* connection env vars from our connection vars ($server, $user, etc.). use use_pg to import $database so it can be different from $database for MySQL
- 02:31 PM Revision 9648: lib/sh/db.sh: added use_pg alias
- 02:31 PM Revision 9647: bugfix: lib/sh/db.sh: psql(): added missing `--set ON_ERROR_STOP=1 --quiet` opts from psql_script_vegbien
- 02:12 PM Revision 9646: lib/sh/db.sh: added psql(), which replaces psql_script_vegbien and psql_verbose_vegbien for general connections. it also supports separate command and stdin files, to allow using `\copy from pstdin`, with pstdin pointing to a separate, EOF-terminated CSV file instead of inlined with the command and terminated with the \. escape (which may be contained within the CSV file itself).
- 01:05 PM Revision 9645: bugfix: lib/sh/local.sh: psql(): $file can't both be passed as a --file param and be prefixed with the necessary \set schema, etc. commands, so instead include $file when cat-ing stdin
- 08:28 AM Revision 9644: added inputs/GBIF/raw_occurrence_record/postprocess.sql, which removes institutions that we have direct data for
- 08:18 AM Revision 9643: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): skip table if already exists (unless remaking), like plant_fraction/make()
- 08:16 AM Revision 9642: bugfix: lib/sh/db.sh: mysql_import(): need to use direct connection to DB instead of via ssh, because ssh does not tunnel nonstandard fds
- 08:15 AM Revision 9641: lib/sh/db.sh: added ssh2local alias
- 07:36 AM Revision 9640: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.plant_fraction.csv_/make(): use new plant_fraction_for_herbaria_filter view
- 07:13 AM Revision 9639: inputs/GBIF/raw_occurrence_record/run: added plant_fraction_for_herbaria_filter/make(). note that for simplicity, plant_fraction_for_herbaria_filter is a view instead of a table.
- 06:50 AM Revision 9638: inputs/GBIF/raw_occurrence_record/run: *.table/*(): renamed to */*() because a target named after a table refers to the table unless it has an explicit file extension
- 06:49 AM Revision 9637: inputs/GBIF/raw_occurrence_record/run: plant_fraction.table/*(): renamed to plant_fraction/*() because a target named after a table refers to the table unless it has an explicit file extension
- 06:41 AM Revision 9636: lib/sh/db.sh: mysql_seal_table(): also revoke GRANT OPTION, which apparently needs to be done in addition (and in a separate command, unlike when granting GRANT OPTION)
- 06:40 AM Revision 9635: lib/sh/db.sh: mysql_seal_table(): REVOKE: ignore errors if REVOKE was already run
- 06:39 AM Revision 9634: lib/sh/db.sh: mysql_seal_table(): REVOKE: removed unneeded explicit database since this is automatically set to the current database
- 06:19 AM Revision 9633: inputs/GBIF/raw_occurrence_record/run: added plant_fraction.table/seal(), which uses new mysql_seal_table()
- 06:19 AM Revision 9632: lib/sh/db.sh: added mysql_seal_table(), which prevents further modifications to a table by a user. this uses new mysql_root().
- 06:18 AM Revision 9631: lib/sh/db.sh: added mysql_root(). this version uses just use_root (compare to the mysql_root() override in local.sh).
- 06:16 AM Revision 9630: lib/sh/local.sh: database connection vars: connect to vegbiendev via ssh and run commands locally, to allow running commands as root (which can only connect to the database locally). this effectively requires an ssh account on vegbiendev, but any ssh account (including an anonymous one, if we set one up) will do. this causes schemas/VegCore/VegCore.my.sql, VegCore.pg.sql to change, because they are now created by mysqldump running on vegbiendev (Linux) instead of on a Mac.
05/29/2013
- 10:35 PM Revision 9629: inputs/GBIF/raw_occurrence_record/run: plant_fraction: added index on plant_fraction for fast extraction of herbaria by fraction threshold
- 10:10 PM Revision 9628: inputs/GBIF/raw_occurrence_record/run: tables: set ENGINE to MyISAM and DEFAULT CHARSET to utf8 to match the other GBIF tables. (note that MyISAM is *not* the default, but is needed to avoid row sort order problems and other issues with InnoDB.)
- 08:09 PM Revision 9627: inputs/GBIF/raw_occurrence_record/run: plant_fraction.table/make(): in remaking mode, drop the table first
- 08:04 PM Revision 9626: inputs/GBIF/raw_occurrence_record/run: plant_fraction.table/make(): only create and populate the table if it doesn't already exist, to avoid clobbering existing data. the noclobber functionality uses new skip_table(), which is the table analog of require_not_exists().
- 08:02 PM Revision 9625: lib/runscripts/table.run, table.run: use new db_make.sh
- 08:02 PM Revision 9624: added lib/sh/db_make.sh that includes both db.sh and make.sh, and will eventually contain DB-related make commands
- 08:00 PM Revision 9623: lib/sh/db.sh: added skip_table(), which prints an already_exists_msg for tables
- 07:56 PM Revision 9622: lib/sh/util.sh: already_exists_msg: undid r9621 because the `|| return 0` should actually always be explicitly specified by the caller, to make it clear that the function will be aborted
- 07:47 PM Revision 9621: lib/sh/util.sh: already_exists_msg(): added alias for use as an error handler. note that ..._not_exists() functions should continue to use the "already_exists_msg" function instead to preserve the exit status.
- 07:40 PM Revision 9620: lib/sh/util.sh: added already_exists_msg() and use it instead of manually generating the die() call
- 07:15 PM Revision 9619: schemas/my.cnf: added innodb_file_per_table so each InnoDB table will get its own file. this should also allow databases with InnoDB tables to be manually renamed.
- 07:09 PM Revision 9618: added schemas/my.cnf from /etc/mysql/my.cnf
- 06:51 PM Revision 9617: schemas/VegCore/VegCore.my.sql, VegCore.pg.sql: synced to VegCore MySQL DB. for some reason, the fkeys are now output in the opposite order from what they were in before.
- 05:22 PM Revision 9616: inputs/.TNRS/schema.sql: MatchedTaxon: filter out rows where Max_score was not high enough to use the TNRS result as a match. removed now-duplicated filter for this in AcceptedTaxon.
- 05:19 PM Revision 9615: inputs/.TNRS/schema.sql: ScrubbedTaxon: removed extra ; at end of WHERE clause
- 03:48 PM Revision 9614: web/links/index.htm: updated to Firefox bookmarks. some broken favicons have also been fixed, by reopening bookmark in Firefox. (this will only update a favicon *if* there is a newer version. to delete a favicon completely, use Firefox's SQLite Manager plugin.)
- 10:17 AM Revision 9613: web/index.php: use XHTML DOCTYPE to match what's used by mod_autoindex. this requires some adjustments in spacing for XHTML's slightly different formatting
- 10:15 AM Revision 9612: bugfix: web/.htaccess: need to do DirectoryIndex redirects *before* checking for existing file/dir, because a DirectoryIndexed dir is existing but still needs to be redirected to the index.* file
- 10:01 AM Revision 9611: web/.htaccess: mod_autoindex: use the main.css stylesheet to match the look-and-feel of index.php
- 10:00 AM Revision 9610: web/.htaccess: mod_autoindex: Note that some listed files are not web-accessible: use ' instead of " to avoid \-escaping embedded "
- 09:39 AM Revision 9609: web/.htaccess: mod_autoindex: sort by description when provided, to allow setting a custom (non-alphabetical) sort order using AddDescription
- 09:37 AM Revision 9608: web/.htaccess: mod_autoindex: added note that some listed files are not web-accessible. they will produce a "Forbidden" error when clicked.
- 09:36 AM Revision 9607: bugfix: web/index.php: added space between the full directory index and the preceding content
- 09:35 AM Revision 9606: web/index.php: moved the full directory index within the rest of the document body
- 08:58 AM Revision 9605: web/index.php: include full directory index, since the URL patterns list is just a subset of the content available through vegpath.org
- 08:25 AM Revision 9604: web/.htaccess: added mod_autoindex IndexOptions, in particular FoldersFirst
- 05:26 AM Revision 9603: bugfix: web/.htaccess: changed "mod_dir listing"->"mod_autoindex listing" because mod_dir does not actually handle the autogenerated listings
- 05:24 AM Revision 9602: bugfix: web/.htaccess: DirectoryIndex: use disabled instead of on because on is actually treated as a filename, and does not invoke mod_autoindex. the DirectoryIndex directive and the mod_dir module actually apply only to manual index files, not to autogenerated dir listings (which are handled by mod_autoindex).
- 04:58 AM Revision 9601: web/index.php: removed no longer needed custom alias j.mp/vegpath# for when page reached through vegbiendev.nceas.ucsb.edu, because vegpath.org is a much more reliable domain than the previous path.vg, and a separate way to reach VegPath when path.vg is down is no longer needed
- 04:43 AM Revision 9600: web/.htaccess: <dir>/all forces mod_dir listing: use simpler $mod_dir_listing env var instead of query string modification to indicate that an explicit mod_dir listing should be displayed. this causes /all to replace ?index=1 as the way to force a mod_dir listing. note that the %{ENV:...} test needs to use $REDIRECT_mod_dir_listing instead of $mod_dir_listing, because a redirect will occur between the /all rule and the index.* rule, causing all env vars to be prepended with REDIRECT_ .
- 03:48 AM Revision 9599: web/.htaccess: <dir>/all forces mod_dir listing, as a simpler syntax than ?index=1
- 03:28 AM Revision 9598: web/.htaccess: for dirs, redirect to index.*: allow requesting a mod_dir listing instead with ?index=1
- 03:26 AM Revision 9597: web/.htaccess: handle DirectoryIndex redirects in a RewriteRule instead of with `DirectoryIndex index`, so that RewriteConds can be used to configure when index.* is used as the DirectoryIndex instead of a mod_dir listing
- 02:30 AM Revision 9596: web/.htaccess: handle DirectoryIndex subrequests when there is no DirectoryIndex: moved comment about -F subrequest after line it applies to
- 02:27 AM Revision 9595: inputs/GBIF/_MySQL/run: documented steps to reload GBIF MySQL
- 02:19 AM Revision 9594: web/.htaccess: RewriteRules: added standard [discardpath,noescape,qsappend] options where missing (these should be the default, but aren't)
05/24/2013
- 03:13 PM Revision 9593: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.table/make(): inline the PRIMARY KEY statement with its column
- 03:10 PM Revision 9592: bugfix: inputs/GBIF/raw_occurrence_record/run: plant_fraction.table/make(): create the table once with "IF NOT EXISTS" and then populate it with INSERT SELECT, to avoid locking it while it's being repopulated. dropping and recreating the table with CREATE TABLE AS prevented phpMyAdmin from even reading the database's tables list, because it was unable to fetch a rowcount for plant_fraction.
- 03:04 PM Revision 9591: lib/sh/db.sh: mysql(): when echoing queries, also echo runtimes (turned on with `--verbose --verbose --verbose`)
- 02:32 PM Revision 9590: added lib/runscripts/datasrc_dir.run
- 02:30 PM Revision 9589: inputs/GBIF/_MySQL/run: added load_data(), which loads the dumpfile into MySQL
- 02:06 PM Revision 9588: lib/sh/db.sh: added mysql_rm_privileged_statements()
- 02:00 PM Revision 9587: bugfix: lib/sh/resume_import.sh: sed calls: moved end-of-line comments to their own line because end-of-line comments are not supported on Mac
- 01:55 PM Revision 9586: lib/runscripts/table_dir.run: renamed table to subdir because this can apply to any datasrc subdir. moved table-specific code to table.run.
- 01:43 PM Revision 9585: lib/runscripts/table_dir.run: renamed table to subdir because this can apply to any datasrc subdir. moved table-specific code to table.run.
- 01:21 PM Revision 9584: lib/runscripts/table_dir.run: table_make(): moved $silent flag to lib/sh/make.sh make() so all make callers can use it
- 12:35 PM Revision 9583: bugfix: inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.run: override ^.preamble.sql/make() and use ../_src/GBIFPortalDB-2013-02-20.dump as the dumpfile instead of this file, which does not contain the preamble
- 12:23 PM Revision 9582: bugfix: lib/sh/resume_import.sh: $preamble_file: use the extension .0.preamble.sql instead of .preamble.sql so the preamble file sorts before the other *.sql files
- 12:22 PM Revision 9581: removed inputs/GBIF/_MySQL/MySQL.data.sql*, since we are using the much faster exported TSVs instead (see raw_occurrence_record/table.tsv). this also avoids confusion between GBIFPortalDB-2013-02-20.data.sql* and MySQL.data.sql* when loading data into MySQL.
- 12:18 PM Revision 9580: bugfix: inputs/GBIF/_MySQL/MySQL.data.sql.run: moved to GBIFPortalDB-2013-02-20.data.sql.run since it's actually the raw input file, not the ANSI export of it, that needs to be imported
- 12:16 PM Revision 9579: lib/sh/resume_import.sh: get_pkey_at_pos(): changed $quote to ` to work with inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql
- 11:50 AM Revision 9578: lib/sh/db.sh: mysql(): added $log_queries flag, which can be turned off to avoid using --verbose. this is useful when running bulk INSERT statements.
- 11:35 AM Revision 9577: lib/sh/local.sh: added mysql_local()
- 11:24 AM Revision 9576: lib/sh/local.sh: added mysql_root()
- 11:24 AM Revision 9575: lib/sh/local.sh: added $root_user, $root_password
- 11:22 AM Revision 9574: lib/sh/db.sh: added use_root alias (similar to use_local/use_remote)
- 11:21 AM Revision 9573: added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.schema.z.clean_up.sql, which removes duplicated and unnecessary indexes in raw_occurrence_record
- 11:20 AM Revision 9572: added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.schema.0.preamble.sql
- 11:02 AM Revision 9571: bugfix: lib/sh/resume_import.sh: sql_preamble(): also stop at first "-- Table structure for table" line (when using a full dumpfile rather than a data-only subset)
- 10:58 AM Revision 9570: lib/sh/resume_import.sh: resume_import(): run connection preamble (first few lines of dumpfile) before continuing with main file at offset, so that connection setting are reapplied
- 06:45 AM Revision 9569: lib/sh/resume_import.sh: is_pkey_imported__int(): use echo_stdout so the user can see the result of the > function in each iteration
- 06:42 AM Revision 9568: added lib/sh/resume_import.sh and use it in inputs/GBIF/_MySQL/MySQL.data.sql.run
- 06:32 AM Revision 9567: inputs/GBIF/_MySQL/MySQL.data.sql.run: is_pkey_imported__int(): made pkey name configurable in $pkey_name
- 05:32 AM Revision 9566: inputs/GBIF/_MySQL/MySQL.data.sql.run: import_resume_pos() run time: removed seconds because the precision is likely only to the nearest half-minute
- 05:31 AM Revision 9565: inputs/GBIF/_MySQL/MySQL.data.sql.run: documented that import_resume_pos() takes 6 min to run, with 37 iterations
- 05:20 AM Revision 9564: added inputs/GBIF/_MySQL/MySQL.data.sql.run, with helper functions for resuming the import to MySQL from where it left off. this is very useful if the import is interrupted for any reason, because otherwise, the entire import would have to be run again from the start, taking 40-50 hours. import_resume_pos() uses new binsearch() to find where in the file the import left off, based on which pkeys have already been imported. (GBIF pkeys are unfortnately not in any order in the input file, nor are they in insertion order in the imported table, because MySQL instead clusters the table by the pkey. this necessitates a much more complex solution to resuming a partial import.)
- 05:14 AM Revision 9563: lib/sh/binsearch.sh: binsearch(): also echo_vars the iter_num, to track how close binsearch is to finding the value (it will always take the same # iters, log2(max - min) )
- 05:11 AM Revision 9562: lib/sh/binsearch.sh: binsearch(): also echo_vars the min/max so these can be used as shortcut inputs if binsearch is run again
- 04:58 AM Revision 9561: bugfix: lib/sh/util.sh: caching: cache_key for function inputs: need to use `declare -p kw_param` instead of "$kw_param" because declare accepts a param name, not value`
- 03:40 AM Revision 9560: lib/sh/binsearch.sh: binsearch(): doc comment: fixed typo in "truncates"
- 03:17 AM Revision 9559: bugfix: lib/sh/util.sh: func_override(): need to match *shortest* __* suffix instead of longest in case the function being overridden itself contained __
- 01:51 AM Revision 9558: bugfix: lib/sh/util.sh: file_size: Linux: need % in %s
- 01:43 AM Revision 9557: lib/sh/db.sh: mysql(): added $data_only flag which enables --skip-column-names and $output_data
- 01:41 AM Revision 9556: bugfix: lib/sh/util.sh: file_size: need to use --format instead of -f on Linux
- 01:22 AM Revision 9555: added lib/runscripts/table_dir.run and use it in table.run
- 01:20 AM Revision 9554: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.ih.csv_/make(): don't use any outer limit value, so that all the IH herbaria are always used. this also ensures that the first GBIF rows will be from an IH herbarium.
- 01:17 AM Revision 9553: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.table/make(): herbaria_filter: don't explicitly set ENGINE or DEFAULT CHARSET, because these should be set to the database values instead so that collations, etc. match
- 12:50 AM Revision 9552: lib/sh/util.sh: filesystem: added file_size alias
- 12:34 AM Revision 9551: lib/sh/util.sh: exceptions: added signals-related functions ignore_sig(), piped_cmd() and helper sig_e()
05/23/2013
- 11:40 PM Revision 9550: lib/sh/util.sh: $sed_cmd: don't use `command`, which causes sed calls (which are usually internal) to always be logged. instead, use echo_run wherever sed needs to be logged.
- 11:38 PM Revision 9549: lib/sh/util.sh: echo_run(): added trailing-space alias to alias-expand next word, which is a command
- 11:31 PM Revision 9548: lib/sh/binsearch.sh: binsearch(): echo $i at log_level 1 so it's displayed by default, as a progress indicator
- 11:30 PM Revision 9547: lib/sh/binsearch.sh: binsearch(): echo $i at log_level 1 so it's displayed by default, as a progress indicator
- 11:29 PM Revision 9546: lib/sh/binsearch.sh: binsearch(): echo the command being run using new echo_run()
- 11:25 PM Revision 9545: lib/sh/util.sh: log+: set PS4 from $log_level instead of relative to its previous value. this allows PS4 to work properly at negative log_levels, in spite of the inability to store a "negative" value in a prefix string.
- 11:23 PM Revision 9544: lib/sh/util.sh: added float_set_min()
- 11:22 PM Revision 9543: lib/sh/util.sh: log+(): log_level: set it using simpler $(()), since log_level will never be fractional (although verbosity can be). log_level may of course be fractional in invoked scripts, but that does not affect util.sh.
- 10:44 PM Revision 9542: lib/sh/util.sh: log++: also track a numeric log_level var, which follows the PS4 prefix
- 10:35 PM Revision 9541: inputs/.TNRS/schema.sql: MatchedTaxon: matchedFamily: use Accepted_family when the Name_matched_accepted_family is not provided, as it's omitted by the current TNRS CSV schema
- 09:54 PM Revision 9540: lib/sh/util.sh: log+(): PS4: split if statement onto multiple lines for clarity
- 09:44 PM Revision 9539: lib/sh/util.sh: added back echo_run(), usable for internal commands where command() would be used for external commands
- 09:33 PM Revision 9538: lib/sh/util.sh: added int2bool()
- 09:25 PM Revision 9537: *{.sh,run}: use new `|| ignore` instead of ignore_e/end_try
- 09:25 PM Revision 9536: lib/sh/util.sh: added ignore(), which uses ||-syntax
- 09:13 PM Revision 9535: lib/sh/util.sh: ignore(): renamed to ignore_e() so ignore() can be used for a simpler, ||-based command
- 09:09 PM Revision 9534: bugfix: lib/sh/util.sh: catch(): need && between test and e=0 so e=0 is only run if $e was equal to the desired value
- 08:22 PM Revision 9533: added lib/sh/binsearch.sh
- 06:27 PM Revision 9532: bugfix: README.TXT: Full database import: screen: need to unset TMOUT, version *after* running `screen` rather than before so they take effect within the `screen` shell
- 06:25 PM Revision 9531: README.TXT: Full database import: after running `screen`: run `set -o ignoreeof` to prevent Ctrl+D from exiting `screen` to keep attached jobs
- 04:40 PM Revision 9530: bin/tnrs_db: documented how to estimate total runtime. note that our tnrs_db wrapper in inputs/.TNRS/tnrs/tnrs.make uses inputs/.TNRS/tnrs/logs/tnrs.make.log.sql as the log file.
- 03:33 PM Revision 9529: inputs/.TNRS/schema.sql, data.sql: updated TNRS CSV columns to preserve Name_matched_accepted_family even though it isn't present in the current TNRS CSVs. this way, Name_matched_accepted_family can still be used for previously-scrubbed names, and family_matched can be added back to analytical_stem_view. (now that bin/tnrs_db uses an explicit columns list in COPY TO, the absence of a column in the CSV is no longer a problem.)
- 03:28 PM Revision 9528: README.TXT: updating TNRS CSV columns: use the entire "COPY tnrs ..." statement instead of just the body of it so that the explicit columns list is included. this way, the COPY statement will cause an error if the TNRS schema was changed but inputs/.TNRS/data.sql was not yet updated.
- 03:00 PM Revision 9527: bin/tnrs_db: removed unused imports
- 02:55 PM Revision 9526: bin/tnrs_db: cumulative_tnrs_profiler: use tnrs.tnrs_request()'s new cumulative_profiler param instead of doing the profiling manually. this also ensures that there isn't extra time between when the cumulative profiler starts/stops and when the per-request profiler starts/stops (because Profiler's new add_subprofiler() method is used).
- 02:53 PM Revision 9525: lib/tnrs.py: single_tnrs_request(): added support for a cumulative profiler using the cumulative_profiler kw param
- 02:53 PM Revision 9524: lib/profiling.py: Profiler: added add_subprofiler(), for use with cumulative profilers
- 02:48 PM Revision 9523: lib/profiling.py: Profiler: added add_time() and use it instead of `self.total +=`
- 02:38 PM Revision 9522: bin/tnrs_db: tnrs_profiler: renamed to cumulative_tnrs_profiler to distinguish it from the tnrs_profiler used by tnrs.tnrs_request(), which just profiles the current request
- 02:36 PM Revision 9521: bugfix: bin/tnrs_db: cumulative profiler: use len(names) instead of this_ct (cur.rowcount) in case the actual # rows fetched differed from the rowcount
- 02:32 PM Revision 9520: lib/tnrs.py: repeated_tnrs_request(): renamed to tnrs_request() since this is the function that should usually be used, to ensure that debugging information is output in the case of an error. (the TNRS request must be made again to output this information.)
- 02:30 PM Revision 9519: lib/tnrs.py: tnrs_request(): renamed to single_tnrs_request() to distinguish it from repeated_tnrs_request()
- 02:25 PM Revision 9518: bin/tnrs_db: removed no longer used $wait flag (which caused tnrs_db to wait max_pause for new rows to be added), because tnrs_db is now invoked automatically after each import by the import_scrub target (in inputs/input.Makefile) and does not need to run as a daemon. note that when scrub is invoked, it is possible that a previous datasource's import has already scrubbed the names for this import, because tnrs_db runs until all rows in tnrs_input_name are scrubbed.
- this also removes clutter in tnrs_db, making it clearer what operations it performs that the library function tnrs.re...
- 02:14 PM Revision 9517: bin/tnrs_db: removed no longer needed explicit population of the Time_submitted, which is now done automatically by the tnrs table. however, this requires starting the transaction *before* submitting data, so Time_submitted is correctly set to the submission time rather than the insertion time. the setting of the correct time can be tested by inserting `time.sleep(n_sec)` after the TNRS request and checking that the Time_submitted is close to the time tnrs_db was run instead of n_sec seconds later.
- 02:09 PM Revision 9516: bin/tnrs_db: start transaction *before* submitting data, so Time_submitted is correctly set to the submission time rather than the insertion time. these may differ by several minutes if TNRS is slow. the setting of the correct time can be tested by inserting `time.sleep(n_sec)` after the TNRS request, removing the explicit setting of Time_submitted, and checking that the Time_submitted is close to the time tnrs_db was run instead of n_sec seconds later.
- 02:05 PM Revision 9515: bugfix: bin/tnrs_db: wrap just the TNRS request and the storing of the response data in a function (undoing part of r9514), because the transaction start time for Time_submitted should not be until the TNRS request is actually made (it often takes several minutes to materialize the next set of input names on a full DB)
- 01:56 PM Revision 9514: bin/tnrs_db: Iterate over unscrubbed verbatim taxonlabels: put loop body in a function (which returns whether or not the loop should continue), so that the loop body can easily be wrapped in a transaction using sql.with_savepoint()
- 01:19 PM Revision 9513: inputs/.TNRS/schema.sql: tnrs.Time_submitted: set default to now() (the timestamp of the start of the current transaction, http://www.postgresql.org/docs/9.1/static/functions-datetime.html) so that it would automatically be populated when rows are added. note that because the start of the current transaction instead of the exact time at insertion is used, all rows inserted in the same transaction (e.g. as part of the same batch) will have the same value for this, linking them together.
- 01:10 PM Revision 9512: inputs/.TNRS/schema.sql: tnrs_populate_derived_fields(): renamed to tnrs_populate_fields() so it can be used to populate other fields as well
- 01:07 PM Revision 9511: bin/tnrs_db: removed no longer needed explicit appending of derived cols, and instead use append_csv()'s new support for importing CSVs whose columns are a subset of the full table
- 12:56 PM Revision 9510: bin/tnrs_db: ColInsertFilters: use the simpler literal value option for the mk_value param
- 12:55 PM Revision 9509: lib/csvs.py: ColInsertFilter: support using a literal value instead of a function for the mk_value param, since this is the most common use case
- 12:43 PM Revision 9508: lib/sql_io.py: append_csv(): support importing CSVs whose columns are a subset of the full table and/or in a different order. when the header exactly matches the columns, the explicit column list will still be omitted as an optimization. this uses code from r4927.
- 12:15 PM Revision 9507: bugfix: lib/runscripts/util.run: need to include sh/make.sh for all runscripts that use make-style commands
- 12:12 PM Revision 9506: *{.sh,run}: use new top_make instead of `make --directory="$top_dir"`
- 12:11 PM Revision 9505: lib/sh/make.sh: added top_make()
- 11:54 AM Revision 9504: inputs/import.stats.xls: Postprocessing: populated entries for analytical DB for last 4 imports, and for backup, backup test for last import. note that the combined import time for the last import is 3.5 days, compared to 3 days for the column-based import portion.
05/22/2013
05/21/2013
- 11:18 PM Revision 9502: inputs/GBIF/Specimen/postprocess.sql, inputs/REMIB/Specimen/postprocess.sql: updated for providers in r9459, which adds TEX
- 11:10 PM Revision 9501: inputs/*/*/postprocess.sql: Remove institutions that we have direct data for: query to obtain list: updated for current schema
- 10:49 PM Revision 9500: inputs/import.stats.xls: Updated import times. GBIF has been refreshed (with the range modeling column subset), and column-based import now takes 3 days for 88.4 million rows.
- 10:27 PM Revision 9499: README.TXT: Full database import: added warning to perform *every single* step listed, to avoid breaking column-based import
- 10:26 PM Revision 9498: README.TXT: Full database import: Publish the new import: added warning to be sure you have done *every single* verification step before proceeding. otherwise, a previous valid import could incorrectly be overwritten with a broken one.
- 09:07 PM Revision 9497: bugfix: README.TXT: Full database import: To run TNRS/remake analytical DB: need to run `export version=<version>` *before* the command which uses it rather than after
- 08:26 PM Revision 9496: added backups/*.md5
- 08:22 PM Revision 9495: added backups/TNRS.2013-5-21.backup.md5
- 07:42 PM Revision 9494: README.TXT: Datasource setup: For MySQL inputs: For .sql exports: added steps to grant privileges to the bien user. the privileges list *excludes* UPDATE, DELETE, ALTER, DROP to prevent bugs in the import scripts from accidentally deleting data.
- 07:37 PM Revision 9493: inputs/.TNRS/schema.sql, data.sql: updated for new TNRS CSV columns (see bug at https://pods.iplantcollaborative.org/jira/browse/TNRS-183). note that these columns may eventually change back (comment by Naim at https://pods.iplantcollaborative.org/jira/browse/TNRS-183#comment-34444).
- 07:33 PM Revision 9492: README.TXT: Full database import: added steps to check that TNRS ran successfully, and fix errors (due to column changes in the TNRS CSV) if it didn't
- 07:24 PM Revision 9491: inputs/test_taxonomic_names/test_scrub: use sh's -e (errexit) mode so errors in an invoked script cause the script to abort instead of burying the error in more output
- 07:19 PM Revision 9490: inputs/test_taxonomic_names/test_scrub: documented that `make schemas/"$public"/uninstall` removes the previous results (since it may be confusing why it's prompting the user to uninstall the schema that is an output of the program)
- 07:16 PM Revision 9489: inputs/test_taxonomic_names/test_scrub: don't need to run the import twice anymore because the accepted names are now included in the tnrs_input_name view that TNRS runs on
- 07:09 PM Revision 9488: inputs/test_taxonomic_names/test_scrub: updated for current TNRS schema
- 06:47 PM Revision 9487: bugfix: inputs/test_taxonomic_names/test_scrub: unset $n so it doesn't limit the # rows. it is set to 2 in the default test environment, so must be unset for n-sensitive programs that should be unlimited.
- 06:40 PM Revision 9486: inputs/test_taxonomic_names/test_scrub: updated for current TNRS schema
- 01:44 PM Revision 9485: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.table/make(): also include the exported plant_fraction herbaria
- 01:43 PM Revision 9484: inputs/GBIF/raw_occurrence_record/run: added herbaria_filter.plant_fraction.csv_/make(), which exports the plant_fraction herbaria whose plant_fraction >= 0.8
- 01:42 PM Revision 9483: inputs/GBIF/raw_occurrence_record/run: added plant_fraction.table/make(), which contains the plant fraction for each herbarium
- 01:37 PM Revision 9482: lib/sh/db.sh: added mk_drop()
- 01:00 PM Revision 9481: lib/sh/util.sh: to_file(): log $stdout so users can tell which file is being created by the command. for some reason, can't use `local redirs=(">$stdout")` because the redirections don't seem to be applied. can't yet use `log+ -2 echo_vars stdout` because log+ does not yet support negative adjustments (they cause PS4 to be emptied out before being re-prepended to).
- 12:54 PM Revision 9480: bugfix: lib/sh/util.sh: log+(): adjustment < 0: need to enclose -$1 in $(()) so it gets evaluated before being used as an array index
- 12:16 PM Revision 9479: lib/sh/local.sh: psql(): documented that --output is actually for query *results*, not echoed statements (and thus must be redirected back to fd 1 while fd 1 with the statements gets sent to the logging port)
- 12:14 PM Revision 9478: lib/sh/local.sh: psql(): documented why can't use fd 11
- 12:09 PM Revision 9477: lib/sh/local.sh: use @redirs instead of manual redirection to set up --output fd, so that the redirection will be echoed along with the command. for some reason, this requires switching to fd 13 instead of 11, because fd 11 gives a "/dev/fd/11: Bad file descriptor" error when 11 is set with exec right before the command instead of on the subshell the command is executed in. (13 was chosen rather than 12 because *2 is for errors, while *3 (or 3*) is for logging.)
- 04:18 AM Revision 9476: bugfix: lib/sh/db.sh: pg_export_table_to_dir_no_header(): inlined $(pg_header) so setting $cols wouldn't affect pg_export_table_no_header(), which uses it as a kw param
05/20/2013
- 10:44 PM Revision 9475: bugfix: lib/sh/util.sh: to_file(): require_not_exists check: missing `test` in `if "$if_not_exists"`
- 10:39 PM Revision 9474: lib/sh/util.sh: command(): log the function call using echo_func to assist debugging. (use a higher log_level because it's internal.)
- 09:29 PM Revision 9473: lib/sh/util.sh: command(): support custom redirections, which will be echoed along with the command
- 08:48 PM Revision 9472: lib/sh/util.sh: to_file(): reworded confusing || conditional for require_not_exists into an if statement
- 08:21 PM Revision 9471: bugfix: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.table/make(): need to use append=1 with mysql_import so the output table doesn't get re-truncated when additional parts are added
- 07:28 PM Revision 9470: bugfix: lib/sh/db.sh: load new aliases before mk_select(), which uses mk_table_esc
- 07:27 PM Revision 9469: lib/runscripts/table.run: include make.sh so runscripts based on it can use make-related utils
- 06:52 PM Revision 9468: lib/sh/db.sh: added mk_select() and use it in mk_select_var
- 06:46 PM Revision 9467: lib/sh/db.sh: added limit() and use it instead of `${limit:+LIMIT $limit}`
- 06:44 PM Revision 9466: lib/sh/db.sh: added mysql_truncate() and use it instead of `mk_truncate|mysql_ANSI`
- 06:42 PM Revision 9465: lib/sh/db.sh: truncate(): renamed to mk_truncate() because it actually just creates a TRUNCATE statement, rather than also executing it
- 06:38 PM Revision 9464: lib/sh/db.sh: use_local/use_remote: unset $prefix after using it so it isn't unintentionally applied as a kw param for a later function
- 04:18 PM Revision 9463: lib/sh/db.sh: mk_select: renamed to mk_select_var since it actually sets a var in the local context rather than returning a query
- 03:40 PM Revision 9462: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.table/make(): specify the different parts used to create the table in an array
- 03:19 PM Revision 9461: inputs/GBIF/raw_occurrence_record/run: renamed herbaria_filter.csv_ to herbaria_filter.ih.csv_ to allow for other tables that get combined into herbaria_filter
- 03:13 PM Revision 9460: bugfix: lib/sh/db.sh: mk_select: ensure newline before LIMIT clause, in case caller provided custom query which did not have trailing newline
05/17/2013
- 06:00 PM Revision 9459: bugfix: mappings/VegCore-VegBIEN.csv: place.geovalid: added missing /1 after _alt
- 05:55 PM Revision 9458: bugfix: lib/sql.py: parse_exception(): typed_name_re: added back matching of names without "", since these are used by some error messages (ones that contain () after the function name)
- 05:41 PM Revision 9457: bugfix: lib/sql.py: parse_exception(): typed_name_re: need to allow " within the matched name, since there are now "" around the entire identifer that was passed to Postgres, which may itself include " . always require "" around the matched name, to ensure that the whole name is matched by .+? e.g. when followed by () for a function call. the version of Postgres we currently use apparently no longer has error messages without the "", so we don't need a separate regexp for quoted and unquoted names.
- 03:43 PM Revision 9456: lib/sh/db.sh: mysql_import(): automatically ensure the table is empty (i.e. using truncate()), unless append=1 is specified. extra calls to truncate() now that this happens automatically have also been removed.
- 01:13 PM Revision 9455: bin/map: by_col: ensure verbosity is at least 2 in live mode (using new ints.set_min() instead of max() for clarity). documented that live column-based import MUST be run with verbosity 2+ (3 preferred) to provide debugging information for often-complex errors. without this, debugging is effectively impossible.
- 01:08 PM Revision 9454: added lib/ints.py with renamings of max()->set_*min*(), min()->set_*max*() for easier understandability of the set-ceiling/set-floor use cases of min()/max()
- 12:57 PM Revision 9453: bin/map: Set default verbosity: by_col: documented that showing all queries is primarily to assist *debugging*, not profiling
- 11:59 AM Revision 9452: lib/sh/util.sh: logging: named it `log++`
- 11:59 AM Revision 9451: lib/sh/util.sh: logging: verbosities: level 0: documented that log++ also suppresses external command output for full support of cron jobs
- 11:57 AM Revision 9450: lib/sh/util.sh: logging: documented `make` equivalents of the various verbosities, where available. (many of the verbosities, such as level 1, are sorely needed in make to avoid excessive output.)
- # verbosities (and `make` equivalents):
# 0: just print errors. useful for cron jobs.
# vs. make: equivalent to --... - 04:03 AM Revision 9449: lib/sh/util.sh: die_e(): benign errors: increase log_level so that a benign non-zero exit status will only be displayed at debug verbosities (2+) (it is confusing otherwise)
- 03:36 AM Revision 9448: lib/sh/util.sh: try(): always run the command with benign_error=1 so that any die_e() doesn't prematurely indicate that a particular exit status was an error
- 03:34 AM Revision 9447: lib/sh/util.sh: die_e(): support benign errors using $benign_error flag that should be logged as info messages instead of errors
- 03:30 AM Revision 9446: lib/sh/util.sh: die(): documented that msg can't use $() (because it would reset $?)
- 03:19 AM Revision 9445: inputs/bien_web/observation/VegBIEN.csv, unmapped_terms.csv: regenerated
- 03:01 AM Revision 9444: lib/sh/util.sh: command(): 2>&$err_fd: add to _redirs *after* echoing command so it isn't echoed at the end of every command (since this redirection is frequently applied)
- 02:55 AM Revision 9443: lib/sh/util.sh: sed: use case statement instead of test to determine flag letter, to easily allow matching multiple `uname` OSes or adding additional flag letters
- 02:46 AM Revision 9442: lib/sh/util.sh: die(): documented that its msg can use $?, because it has not yet been overridden by another command
- 02:45 AM Revision 9441: lib/sh/util.sh: die_e(): use die(), which performs the necessary save_e/rethrow. this requires using $? instead of $e for the exit status, because $e has not yet been set.
- 02:42 AM Revision 9440: lib/sh/util.sh: inlined log_e() into die_e() because that's the only place it's used
- 02:37 AM Revision 9439: lib/sh/util.sh: command(): print "command exited with error" message using new die_e() if command returns false. this requires removing manual die_e()/log_e() calls elsewhere.
- 02:34 AM Revision 9438: lib/sh/util.sh: command(): moved increase of indent inside () so that error-handling statements after () will use the outer log_level
- 02:31 AM Revision 9437: lib/sh/util.sh: added die_e(), which logs that a command exited with an error
- 02:18 AM Revision 9436: lib/sh/util.sh: command(): determine redirections before echoing the command so they can be logged along with the command, instead of as separate exec statements. (these had a higher log_level to avoid cluttering the output with `exec` lines, which usually suppressed the redirections completely.) inline the command__set_fds() nested func so the redirections are all in one place.
- 01:54 AM Revision 9435: lib/sh/util.sh: use simpler `if can_log; then indent; fi` instead of `can_log && indent || true`. however, the `&& indent || true` syntax is still required in aliases such as echo_func which need to allow prefixing the command with a wrapper command or kw param assignments.
05/16/2013
- 09:28 PM Revision 9434: inputs/GBIF/raw_occurrence_record/run: dynamically generate herbaria_filter.csv_ from herbaria.ih in new target herbaria_filter.csv_/make()
- 09:27 PM Revision 9433: inputs/GBIF/raw_occurrence_record/run: store the herbaria filter in a MySQL table loaded from a CSV instead of getting it from a hardcoded list of IN (...) values
- 09:24 PM Revision 9432: lib/sh/db.sh: added truncate()
- 09:23 PM Revision 9431: lib/sh/make.sh: set_make_vars: set $target_stem
- 08:49 PM Revision 9430: lib/sh/db.sh: added mysql_import()
- 07:02 PM Revision 9429: lib/sh/db.sh: removed no longer used mk_esc_name()
- 07:01 PM Revision 9428: lib/runscripts/table.run: don't mk_esc_name schema, table because these will be mk_esc_name'd by functions that use them
- 06:55 PM Revision 9427: lib/sh/local.sh: psql(): use $schema_esc, $table_esc instead of just putting $schema, $table in ""
- 06:48 PM Revision 9426: lib/sh/db.sh: mk_esc_name_alias(): don't overwrite an already-defined $*_esc, to allow the user to provide an already-escaped value (such as a schema-qualified table) directly
- 06:38 PM Revision 9425: lib/sh/util.sh: rtrim(): increase the log_level of sed to 4+ instead of 2+ because it is usually run as part of a var assignment, and should therefore have a lower log_level than echo_vars
- 06:32 PM Revision 9424: lib/sh/db.sh: mk_esc_name_alias(): echo_vars the *_esc var when it's set
- 06:31 PM Revision 9423: lib/sh/db.sh: added mk_esc_name_alias() and use it to create mk_schema_esc, mk_table_esc
- 05:55 PM Revision 9422: lib/sh/db.sh: mysql(): run with --local-infile=1
- 05:48 PM Revision 9421: bugfix: lib/sh/db.sh: log_sql(): use can_log() instead because the verbosity now gets decremented as the log_level increases, so the threshold to compare to is 0 instead of 2
- 05:46 PM Revision 9420: lib/sh/util.sh: added set_default()
- 05:45 PM Revision 9419: lib/sh/util.sh: rtrim(): run at higher log_level so that sed command is not normally echoed
- 04:40 PM Revision 9418: inputs/GBIF/raw_occurrence_record/run: renamed herbaria.sql to herbaria.data.sql so it wouldn't be added to svn by `make inputs/GBIF/raw_occurrence_record/add` or `make inputs/add`
- 04:38 PM Revision 9417: inputs/input.Makefile: $(svnFiles): also exclude *.data.sql, which should never be in svn
- 04:27 PM Revision 9416: schemas/vegbien.sql: cultivated_family_locations: documented that table is from sftp://nimoy.nceas.ucsb.edu/home/bien/bien2_scripts/geoscrub/cultivated/cult_by_taxon/flag_by_taxa.inc (i.e. not generated by a function)
- 04:15 PM Revision 9415: schemas/vegbien.sql: place.geovalid: added latLongDomainValid to the values to _and together
- 04:09 PM Revision 9414: schemas/vegbien.sql: place.geovalid: require it to be NOT NULL so that it's always a 2-valued boolean (but default it to false since it's not a required field)
- 04:06 PM Revision 9413: mappings/VegCore-VegBIEN.csv: place.geovalid: use false instead of NULL
- 03:46 PM Revision 9412: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): exclude deleted rows (i.e. where the deleted timestamp is non-NULL)
- 03:42 PM Revision 9411: inputs/GBIF/raw_occurrence_record/header.csv: regenerated using ./run. since the table is reimported as a CSV, it uses bin/csv2db, which prepends an additional row_num column.
- 03:09 PM Revision 9410: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): remove explicit cols list to include all cols. the file size of the generated table.tsv will increase by ~3x, but should remain reasonably-sized compared to our available disk space.
- 03:04 PM Revision 9409: bugfix: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): need \ line continuation after vars so they only apply to the command rather than being set as global vars
- 03:02 PM Revision 9408: bugfix: lib/runscripts/table.run: load_data(): use new $verbosity_min instead of running `verbosity_min` so that the command name logging is not output with the new verbosity
- 02:59 PM Revision 9407: lib/sh/util.sh: added $verbosity_min to set a `verbosity_min` value *after* the command name, etc. has been logged, so that the logging itself is not output with the new verbosity
- 02:38 PM Revision 9406: schemas/vegbien.sql: range_modeling_input: include only plants (i.e. rows with higher_plant_group IS NOT NULL)
- 02:36 PM Revision 9405: schemas/vegbien.sql: range_modeling_input: added higher_plant_group, for use in restricting rows to plants
- 02:24 PM Revision 9404: inputs/.geoscrub/geoscrub_output/map.csv: *validity: added definitions of the numeric codes from _src/README.TXT
- 01:11 PM Revision 9403: added planning/workflow/validation/GeoDistKM.sql.txt
- 12:42 PM Revision 9402: planning/goals/BIEN3_derived_data_products.docx: updated to most recent version from Brad's e-mail on 2013-4-16
- 11:18 AM Revision 9401: /README.TXT: Full database import: before running screen: added `unset TMOUT` because TMOUT (autologout) causes screen to exit even with background processes active
- 11:17 AM Revision 9400: /README.TXT: Maintenance: added things to put in your .profile on a live machine (e.g. vegbiendev). in particular, you *MUST NOT* have a TMOUT (autologout) set, because this causes screen to exit even if background processes (e.g. from column-based import) are running
- 11:15 AM Revision 9399: /README.TXT: Maintenance: added things to put in your .profile on a live machine (e.g. vegbiendev). in particular, you *MUST NOT* have a TMOUT (autologout) set, because this causes screen to exit even if background processes (e.g. from column-based import) are running
- 06:15 AM Revision 9398: bugfix: lib/runscripts/table.run: load_data(): need to ensure the verbosity is at least 3 because the install logs require verbose output. (3 is the default for the installer, but is overridden by the runscripts, which instead set the default to 1.)
- 06:13 AM Revision 9397: lib/sh/util.sh: logging: added verbosity_min()
- 06:00 AM Revision 9396: lib/sh/util.sh: logging: added verbosity_int() and use it instead of `round_down "$verbosity"`
- 02:08 AM Revision 9395: bugfix: lib/sh/util.sh: log+ alias: don't expand next word because it's not a cmd
- 01:59 AM Revision 9394: lib/runscripts/table.run: $schema, $table: log the `cd` used to calculate the value at log_level 3 instead of 1 (note that the cd() function call for this will be logged at log_level 5)
- 01:51 AM Revision 9393: lib/sh/util.sh: cd(): trace-log the cd() function call itself (at log_level 3) in addition to the cd builtin call
Also available in: Atom