inputs/.TNRS/schema.sql: tnrs_populate_fields(): Is_plant: must match family as Family_score = 1 (as discussed during conference call vegpath.org/wiki/2013-05-30_conference_call#postprocess-TNRS-results-to-exclude-animals-with-genus-homonyms) instead of as Family_matched IS NOT NULL (as listed in Brad's formula at vegpath.org/wiki/Result_filtering#TNRS-results) because TNRS transforms animal to plant families via fuzzy matching, necessitating a Family_score check to ensure an exact match to a plant family that was not transformed from an animal family

inputs/.TNRS/schema.sql: added Is_plant derived field, which is populated using the formula at vegpath.org/wiki/Result_filtering#TNRS-results . note that the homonym filtering is currently excluded until we determine whether we can get direct access to the IRMNG homonyms database (http://www.cmar.csiro.au/datacentre/irmng/homonyms.htm). note also that changes to the TNRS schema cannot be fully tested until any TNRS client bugs are fixed, because the data.sql updater requires a working TNRS client to regenerate the sample data.

inputs/.TNRS/schema.sql: updated for current TSV schema: renamed Accepted_species->Accepted_name_species, Accepted_family->Accepted_name_family

bugfix: schemas/vegbien.sql: tnrs_input_name: must anti-join against MatchedTaxon rather than ValidMatchedTaxon to ensure that all of TNRS.tnrs is excluded from the input names. this prevents duplicates from appearing in the TNRS results, which would break the TSV import into TNRS.tnrs. it also prevents no-match names from being scrubbed repeatedly because they were not properly filtered out of the input names.

inputs/.TNRS/schema.sql: fixed whitespace

inputs/.TNRS/schema.sql: added MatchedTaxon view, which now just renames the columns but does not filter the results, and use it in ValidMatchedTaxon

inputs/.TNRS/schema.sql: MatchedTaxon: renamed to ValidMatchedTaxon since this view actually contains only the names with a valid match

bugfix: lib/sql.py: parse_exception(): make_DuplicateKeyException(): handle nested exceptions (which should never be generated, but may be in case of sql.py bugs such as wiki.vegpath.org/To_Do#Fixes > #1) by printing the nested exception and then rethrowing the original exception, so that the original exception does not get lost and still ends up at the end of the program's output, to enable debugging

inputs/.TNRS/schema.sql: tnrs: documented that when changing this table's schema, you must regenerate data.sql using `inputs/test_taxonomic_names/test_scrub`

inputs/GBIF/raw_occurrence_record/run: table.tsv.gz/make(): documented runtime (35 min)

bugfix: schemas/vegbien.sql: analytical_stem_view: speciesBinomialWithMorphospecies: if accepted name not specified, use matched name (matched*) or Name_submitted (concatenatedScientificName), as described at http://wiki.vegpath.org/2013-05-30_conference_call#fix-TNRS-speciesBinomialWithMorphospecies-to-include-alternatives-when-no-accepted-name

lib/sh/make.sh: make(): time all invocations of make

lib/sh/make.sh: make() at verbosity < 4, hide messages about making included Makefiles: use sed with a range expression (/.../,/.../) to also exclude all log messages between an opening "make ...Makefile" and a closing "make[#]: ...Makefile"

lib/sh/util.sh: log+ aliases: added clog++/-- aliases for cmds, which don't include log_local. these are useful when you can't just use "log++" because you need the command following it to be alias-expanded.

bugfix: lib/sh/make.sh: make(): use [:char_class:] exprs instead of \X char class abbrs because the \ abbrs are not supported on Linux

inputs/GBIF/table.run: table.tsv/make(): remake table.tsv.gz/make() after table.tsv is made

inputs/GBIF/table.run: added table.tsv.gz/make()

lib/sh/util.sh: added bg_cmd() and helpers log_bg(), log_last_bg()

lib/runscripts/subdir.run: auto-append -remake to all targets in remake mode

lib/runscripts/subdir.run: subdir_make(): put prepending of "$subdir/" on its own line for clarity

bugfix: lib/common.Makefile: gzip/gunzip: in `touch -r`, $< needs to be the reference file

*{.sh,run}: make() calls: removed no longer applicable silent=1, which is now handled automatically by the log_level mechanism

/Makefile: new config target (part of install): install our bin/make in the user's ~/bin dir so that bin/make (with filtering of verbose messages) will always be used instead of standard make (without us needing to change every occurrence of make to bin/make!)

bugfix: bin/make: use standard make logging port (1) instead of $log_fd (30) so that the output of bin/make can in turn be filtered by util.sh using the standard cmd_log_fd=1

lib/sh/util.sh: command(): print the log_fd in effect in case the user changes it

lib/sh/make.sh: make(): log the messages about making included Makefiles at log_level 4 instead of 3, so that verbosity 3 can be used to see the vars being used by the filtering process (i.e. the filtering process must still be enabled at verbosity 3)

bin/make: use `readlink -f` on BASH_SOURCE⁰ so that this script can also be run via a symlink

lib/sh/util.sh: no_PATH_recursion(): also remove $top_symlink_dir_abs

lib/sh/util.sh: added $top_symlink_dir_abs

lib/sh/util.sh: $top_dir_orig: renamed to $top_symlink_dir since this reflects better what the dir actually represents, and when it differs from $top_dir

lib/sh/make.sh: make(): fixed func name in usage

bin/make: don't print make cmd by default, so that only `make` output is printed at verbosity 1

bin/make: don't reinvoke make() if the make filter has already been set up, as indicated by $is_outermost (instead, invoke make directly using exec)

lib/sh/util.sh: added $is_outermost, which stores whether the util.sh env has not yet set up

lib/sh/util.sh: echo_redirs_cmd(): added $cmd_name_log_inc kw param to allow echoing the command name at a different log_level than the command output

lib/sh/make.sh: make(): log the filter setup operations at log_level 2 so they do not clutter up normal output

lib/sh/util.sh: filter_fd: alias-expand filter_cmd

added bin/make, which runs make, hiding verbose messages about making included Makefiles. this should be used in preference to plain make, to avoid excessive log messages that prevent the user from seeing the core commands that are being run.

lib/sh/util.sh: added no_PATH_recursion(), which allows running a system command of the same name as the script

lib/sh/util.sh: added PATH_rm(), which removes components from the PATH

lib/sh/util.sh: added $top_dir_abs, $top_dir_orig

bugfix: lib/sh/util.sh: command(): use `builtin command` instead of `exec` so that options like -p (reset PATH) work properly. also, the command builtin it overrides is designed to be used with more than just external commands, and command() should not impose additional limitations.

lib/sh/util.sh: added self_sys alias, which uses only system utilities (`command -p`) instead of the current PATH

lib/sh/make.sh: make(): at verbosity < 3, hide messages about making included Makefiles. this makes the make output much more readable when a Makefile contains an include statement, because there won't be a ton of log messages every time a Makefile is included. this filtering is so useful that it probably makes sense to run make for any of our Makefiles using `lib/runscripts/util.run make ...` instead of plain make. compare e.g. `make inputs/ACAD/Specimen/map.csv` (53 lines of output) and `lib/runscripts/util.run make inputs/ACAD/Specimen/map.csv` (17 lines of output, 1/3 as much).

lib/sh/util.sh: log++: before cmd: documented that you need to use "log++" instad of log++ to avoid using the log++ alias, which prepends a log_local call. omitting the quotes is generally not a problem, but when there is another command wrapping the log++, you need the "" to avoid the wrapper applying to log_local's declare call instead.

lib/sh/util.sh: log++: before cmd: documented how to use it as `log+ #` when incrementing multiple log_levels at once (this is a better method than `"log++" "log++" ...`)

lib/sh/util.sh: log++: before cmd: documented that you need to use it as `"log++" "log++"` when repeating it multiple times

lib/sh/util.sh: added filter_fd(), which encapsulates the use of >() process substitution for filtering an fd other than stdout (yes, this is possible without lots of 3>&1 1>&2 2>&3 redirections!). this can be useful e.g. to filter logging output or highlight errors.

lib/sh/util.sh: log+(): documented that with a cmd, assignments are applied just to the cmd, so log_local is not needed

lib/sh/util.sh: moved pipe_delay() before fd-related functions so it can be used by them

lib/sh/util.sh: removed no longer needed load new aliases before echo_stdin(), since pipe_delay() is now a function

bugfix: lib/sh/util.sh: set_fds(): add #<>&- before every #<>&# reopen: need to use loop var $i instead of $1 (which would have been used with a while/shift method of iterating over $@)

bugfix: lib/sh/util.sh: set_fds(): added workaround for strange bash bug where reopening an fd sometimes first requires explicitly closing it, by adding an <>&- entry for every redirection

lib/sh/util.sh: added match_prefix()

lib/sh/make.sh: make(): determine --silent status based on the verbosity (<=0) instead of a kw param

inputs/GBIF/table.run: table.tsv.md5/make(): only use %-remake if remaking

lib/sh/make.sh: make(): removed extra space after --silent

lib/sh/db.sh: moved mysql_root() after the mysql->mysql_ANSI alias (and load new aliases) so that it will also use ANSI mode and support "" identifiers

lib/sh/db.sh: mysql: always use ANSI mode, to support "" identifiers. note that `` are still supported in this mode, so it also works with SHOW CREATE TABLE output and dumpfiles.

bugfix: inputs/GBIF/raw_occurrence_record/run: plant_fraction_for_herbaria_filter/make(): need to make prerequisites first (plant_fraction/make)

bugfix: inputs/GBIF/table.run: table.tsv.md5/make(): use %-remake to ensure that the .md5 file is remade, regardless of the .md5 file's mtime relative to table.tsv. you would generally expect table.tsv's new mtime to be newer than the .md5 file's (thus triggering make to run), but if you e.g. ran svn up after making the table.tsv, this might not be the case.

/Makefile: moved %/remake, %-remake to lib/common.Makefile because they are generally useful

/Makefile: moved %/reinstall to lib/common.Makefile because it is generally useful

lib/sh/db.sh: mysql_cmd(): run the command with `time` because in mysql()'s output_data mode, no queries, and therefore no runtimes, are echoed, so the total runtime needs to be echoed separately instead. the total runtime is also useful in general, when many long-running queries are run and you would also like to know the total time (e.g. in make_analytical_db).

lib/sh/db.sh: limit(): usage: surrounded query in "" to clarify that it's a string, not a command

inputs/GBIF/table.run: table.tsv/make(): use new set_large_table to prevent table.tsv from being deleted on error for full export runs (while still deleting it on error for sample subset runs)

lib/sh/db.sh: added set_large_table alias, used to set to_file's $del flag based on $limit

lib/sh/util.sh: added exit2bool()

lib/sh/util.sh: int2bool(): renamed to int2exit() because it actually sets a boolean exit status rather than returning a boolean string

bugfix: lib/sh/util.sh: to_file(): also kw_params the other kw params if_not_exists, del

lib/sh/db.sh: mysql_cmd(): use --quick to avoid buffering entire result (which is slow and memory-intensive for large result sets). this option applies to both mysql() and mysqldump().

lib/sh/make.sh: self_make(): documented that it should be preceded by set_make_vars

bugfix: inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.run: ^.preamble.sql/make(): need to run set_make_vars even though the make vars are not used, because set_make_vars sets $_remake, which is needed by self_make

lib/sh/make.sh: set_make_vars: echo_vars $_remake to help debugging

bugfix: lib/sh/util.sh: to_file(): need to run invoked cmd using redir so that >$stdout redir is applied properly when cmd is a shell function instead of an external command (external commands were already redirected properly because command() calls redir). this fixes a bug in inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.run ^.preamble.sql/make(), where the generated file would be output to stdout instead of to the file because to_file()'s command was a shell function, and therefore the redirection was not applied. this fix requires redir() to be a separate function from command(), because command() does many things that are only applicable to external commands.

lib/sh/util.sh: redir(): override save_e and add `unset redirs` so error handlers are not redirected

lib/sh/util.sh: added alias_append(), similar to func_override() for aliases

lib/sh/util.sh: redir(): don't do redir actions if redir will be run later (i.e. if command to run starts with `redir` or `command`)

lib/sh/util.sh: redir(), command(): log their function calls at the usual log_level (2) instead of one higher, so that they are printed in the call tree to help debugging

lib/sh/util.sh: added redir() and use it in command() to perform and echo the redirections

lib/sh/util.sh: command(): removed comment that "the following redirections must happen in exactly this order", because there is now only one redirection

lib/sh/util.sh: command(): removed 2>&$err_fd, which is no longer needed because $err_fd is now 2 (in general, there is probably not a need for a special $err_fd var, because 2 is already stderr)

lib/sh/util.sh: command(): don't set err_fd to global stderr, because this prevents errors from being captured and parsed by callers. it is also not necessary to use a separate port than stderr, because stderr already contains only errors (since logging messages go to their own port). global stderr would still be useful e.g. for displaying input prompts the user when reading from global stdin.

bugfix: *run: overriding targets: use new self_make to properly progagate the $remake flag to the overridden target, so that the target itself is not skipped

lib/sh/util.sh: to_file(): added del= flag to prevent the file from being auto-removed on error (e.g. to preserve a partial result, which normally would be removed)

lib/sh/make.sh: added self_make(), which progagates the $remake flag (normally it is not progagated, because prerequisites should not also be remade)

lib/sh/util.sh: usage comments: when there is a descriptive comment on the same line as the usage, prepend it with # (as if it were an end-of-line comment) instead of enclosing it in (), to make it visually obvious that it's a comment and not part of the usage commands

lib/sh/util.sh: added all_funcs(), which lists all declared functions

bugfix: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added check_target_exists so table.tsv would not be overwritten if it already existed

bugfix: inputs/GBIF/table.run: table.tsv/make(): force table.tsv.md5 to be remade (using remake=1) because the table.tsv contents will have changed

bugfix: lib/sh/make.sh: set_make_vars: can't use end-of-line comment in alias because it will comment out whatever is after the alias where it's used. can't just put a newline or ; after the end-of-line comment because the alias's lines will be combined onto one line using ; , so end-of-line comments would not be supported.

lib/sh/util.sh: added pf(), to print a function declaration for debugging

inputs/GBIF/raw_occurrence_record/run: plant_fraction/make(): documented runtime (1 hr)

*.url: mailto URLs: use the standard e-mail dotpath syntax e-mail@host?name.date.subject.(attachment) (vegpath.org/wiki/Global_IDs#Resource). populated missing fields (e.g. name, subject) where needed.

*.url: mailto URLs: ensured that they are proper URLs with the mailto: protocol

lib/sh/util.sh: to_file(): exc handler that rm's file: unset redirs so it isn't used in the rm cmd

inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): removed no longer needed explicit clear of $remake, which is now done by make.sh instead

lib/sh/make.sh: set_make_vars: don't progagate remake to prerequisites, so that remake=1 only applies to the outermost target rather than forcing every prerequisite to be remade, too

lib/sh/make.sh: moved remaking section before set_make_vars so that it can be used in set_make_vars

inputs/GBIF/raw_occurrence_record/run: added herbaria_filter/seal()