Project

General

Profile

Statistics
| Revision:

# Date Author Comment
9797 06/09/2013 06:05 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added echo_vars() stub

9796 06/09/2013 05:43 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added echo_run() stub

9795 06/09/2013 05:39 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: set_fds(): don't run (or echo) exec if no redirections are being made

9794 06/09/2013 05:37 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: added missing stub for indent alias (used by echo_func alias, which is a stub). without the stub, /usr/bin/indent would be used instead on Mac.

9793 06/09/2013 04:01 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: root_rel_path(): added echo_func

9792 06/09/2013 04:01 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: root_rel_path(): use canon_rel_path instead of rel_path because $1 may be absolute rather than relative to the current dir, so $root_dir needs to be absoluted (which requires $1 to be absoluted as well)

9791 06/09/2013 03:56 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: support custom $base_dir, which will be run through realpath() to match $path ($PWD, which was used before, did not need to be realpath'd because it was already absolute)

9790 06/09/2013 03:34 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved echo_func alias to stub because it must be embedded in its expanded alias form to work properly

9789 06/09/2013 03:29 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: declare echo_func as a stub before it's defined, so that functions can use it even if they are defined before it (and its logging functionality will be enabled as soon as it's defined)

9788 06/09/2013 03:26 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: rel_path(): don't log++ it, and instead only log++ applicable calls of it or its callers. this allows non-internal calls of rel_path() to be logged at the usual log_level.

9787 06/09/2013 01:00 PM Aaron Marcuse-Kubitza

lib/sh/local.sh: added root_rel_path()

9786 06/09/2013 12:39 PM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: unescape HTML in page's description, such as links to more info. this is necessary to properly render the persistent shells link in the screen > scrollback folder description.

9785 06/09/2013 12:37 PM Aaron Marcuse-Kubitza

bin/repl: added unescape_html() filter function, which can be specified as the replacement string

9784 06/09/2013 12:35 PM Aaron Marcuse-Kubitza

bin/repl: support Unicode characters in the matched portion of the string

9783 06/09/2013 11:52 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. `screen`: scrollback: added link to our wiki page on persistent shells, which are a better way to support reconnecting.

9782 06/09/2013 11:36 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. added bookmarks about the `screen` command, especially how to access the scrollback. resorted several folders alphabetically.

9781 06/09/2013 11:13 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): documented how to view the runtime when using `screen` (press Ctrl-A [ , use up-arrow, and then press Esc to leave copy mode)

9780 06/09/2013 11:12 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record_plants/run: herbaria_filter/make(): use new ih_herbarium table instead of the herbaria_filter.ih.csv_ file directly

9779 06/08/2013 12:23 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record_plants/run: added ih_herbarium/make(), which stores the IH herbaria

9778 06/08/2013 11:50 AM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/raw_occurrence_record_plants/run: table/make(): also filter out rows with a non-plant family (as described at http://vegpath.org/wiki/2013-06-06_conference_call#GBIF-subsetting-fix-raw_occurrence_record-filter-formula), since some institutions have both animal and plant rows, even though they are in IH or in the 80% list. (note that NULL families are OK.)

9777 06/08/2013 04:12 AM Aaron Marcuse-Kubitza

*{.sh,run}: use mysql instead of mysql_ANSI because mysql is now an alias to mysql_ANSI (since ANSI mode still supports key MySQL features, like `` quotes)

9776 06/08/2013 04:09 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): documented that incremental output is provided right away with --quick (unbuffered), but takes awhile to become visible in Macfusion sshfs. this can be tested with `while true; do stat inputs/GBIF/raw_occurrence_record_plants/table.tsv; sleep 2; done` running concurrently with `./inputs/GBIF/raw_occurrence_record_plants/run table.tsv/make` on vegbiendev:/home/bien/svn .

9775 06/08/2013 04:00 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): use new raw_occurrence_record_plants view from table/make()

9774 06/08/2013 03:15 AM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/raw_occurrence_record_plants/run: table/make(): added make of prerequisites

9773 06/08/2013 03:14 AM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/raw_occurrence_record_plants/run: table/make(): don't reset $table to plant_fraction_for_herbaria_filter for commands that use $table

9772 06/08/2013 03:10 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record_plants/run: added table/make(), which makes the filter view

9771 06/08/2013 02:14 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/: renamed to raw_occurrence_record_plants because it's actually only the plants in raw_occurrence_record, not all of raw_occurrence_record. also, this will allow us to create a separate raw_occurrence_record_plants view whose name matches the folder and does not collide with the raw_occurrence_record table.

9770 06/08/2013 12:44 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): added runtime, which is ~0 since it just needs to do CSV import and index scans

9769 06/08/2013 12:43 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): time the population of herbaria_filter

9768 06/07/2013 11:47 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: plant_fraction/make(): updated runtime. added rows affected count to runtime so if the number of rows it's related to (in this case, institution_code) changes, the runtime can be expected to change accordingly.

9767 06/06/2013 04:54 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs_populate_fields(): documented runtime (17 min)

9766 06/06/2013 04:49 PM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/raw_occurrence_record/run: plant_fraction/make(): plant_fraction column: COUNT counts non-NULL rather than true values (which counter-intuitively includes false, because it's non-NULL), so need to add NULLIF around the boolean expression to turn it into a NULL-or-not expression. see http://vegpath.org/wiki/2013-06-06_conference_call#GBIF-subsetting-fix-plant_fraction-SQL-bug .

9765 06/06/2013 04:29 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs_populate_fields(): documented that when changing this function, you must regenerate the derived cols using `UPDATE tnrs SET "Name_submitted" = "Name_submitted"`

9764 06/06/2013 04:21 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs_populate_fields(): Is_plant: must match family as Family_score = 1 (as discussed during conference call vegpath.org/wiki/2013-05-30_conference_call#postprocess-TNRS-results-to-exclude-animals-with-genus-homonyms) instead of as Family_matched IS NOT NULL (as listed in Brad's formula at vegpath.org/wiki/Result_filtering#TNRS-results) because TNRS transforms animal to plant families via fuzzy matching, necessitating a Family_score check to ensure an exact match to a plant family that was not transformed from an animal family

9763 06/06/2013 03:29 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: added Is_plant derived field, which is populated using the formula at vegpath.org/wiki/Result_filtering#TNRS-results . note that the homonym filtering is currently excluded until we determine whether we can get direct access to the IRMNG homonyms database (http://www.cmar.csiro.au/datacentre/irmng/homonyms.htm). note also that changes to the TNRS schema cannot be fully tested until any TNRS client bugs are fixed, because the data.sql updater requires a working TNRS client to regenerate the sample data.

9762 06/06/2013 02:42 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: updated for current TSV schema: renamed Accepted_species->Accepted_name_species, Accepted_family->Accepted_name_family

9761 06/06/2013 02:27 PM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: tnrs_input_name: must anti-join against MatchedTaxon rather than ValidMatchedTaxon to ensure that all of TNRS.tnrs is excluded from the input names. this prevents duplicates from appearing in the TNRS results, which would break the TSV import into TNRS.tnrs. it also prevents no-match names from being scrubbed repeatedly because they were not properly filtered out of the input names.

9760 06/06/2013 02:17 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: fixed whitespace

9759 06/06/2013 02:15 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: added MatchedTaxon view, which now just renames the columns but does not filter the results, and use it in ValidMatchedTaxon

9758 06/06/2013 02:11 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: MatchedTaxon: renamed to ValidMatchedTaxon since this view actually contains only the names with a valid match

9757 06/06/2013 12:59 PM Aaron Marcuse-Kubitza

bugfix: lib/sql.py: parse_exception(): make_DuplicateKeyException(): handle nested exceptions (which should never be generated, but may be in case of sql.py bugs such as wiki.vegpath.org/To_Do#Fixes > #1) by printing the nested exception and then rethrowing the original exception, so that the original exception does not get lost and still ends up at the end of the program's output, to enable debugging

9756 06/06/2013 11:08 AM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: tnrs: documented that when changing this table's schema, you must regenerate data.sql using `inputs/test_taxonomic_names/test_scrub`

9755 06/06/2013 08:09 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv.gz/make(): documented runtime (35 min)

9754 06/06/2013 07:52 AM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: analytical_stem_view: speciesBinomialWithMorphospecies: if accepted name not specified, use matched name (matched*) or Name_submitted (concatenatedScientificName), as described at http://wiki.vegpath.org/2013-05-30_conference_call#fix-TNRS-speciesBinomialWithMorphospecies-to-include-alternatives-when-no-accepted-name

9753 06/06/2013 07:24 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: make(): time all invocations of make

9752 06/06/2013 07:14 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: make() at verbosity < 4, hide messages about making included Makefiles: use sed with a range expression (/.../,/.../) to also exclude all log messages between an opening "make ...Makefile" and a closing "make[#]: ...Makefile"

9751 06/06/2013 06:35 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: log+ aliases: added clog++/-- aliases for cmds, which don't include log_local. these are useful when you can't just use "log++" because you need the command following it to be alias-expanded.

9750 06/06/2013 05:52 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/make.sh: make(): use [:char_class:] exprs instead of \X char class abbrs because the \ abbrs are not supported on Linux

9749 06/06/2013 05:33 AM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make(): remake table.tsv.gz/make() after table.tsv is made

9748 06/06/2013 05:27 AM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: added table.tsv.gz/make()

9747 06/06/2013 05:26 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added bg_cmd() and helpers log_bg(), log_last_bg()

9746 06/06/2013 05:04 AM Aaron Marcuse-Kubitza

lib/runscripts/subdir.run: auto-append -remake to all targets in remake mode

9745 06/06/2013 04:15 AM Aaron Marcuse-Kubitza

lib/runscripts/subdir.run: subdir_make(): put prepending of "$subdir/" on its own line for clarity

9744 06/06/2013 04:00 AM Aaron Marcuse-Kubitza

bugfix: lib/common.Makefile: gzip/gunzip: in `touch -r`, $< needs to be the reference file

9743 06/06/2013 02:36 AM Aaron Marcuse-Kubitza

*{.sh,run}: make() calls: removed no longer applicable silent=1, which is now handled automatically by the log_level mechanism

9742 06/06/2013 02:35 AM Aaron Marcuse-Kubitza

/Makefile: new config target (part of install): install our bin/make in the user's ~/bin dir so that bin/make (with filtering of verbose messages) will always be used instead of standard make (without us needing to change every occurrence of make to bin/make!)

9741 06/06/2013 02:25 AM Aaron Marcuse-Kubitza

bugfix: bin/make: use standard make logging port (1) instead of $log_fd (30) so that the output of bin/make can in turn be filtered by util.sh using the standard cmd_log_fd=1

9740 06/06/2013 02:23 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: command(): print the log_fd in effect in case the user changes it

9739 06/06/2013 02:23 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: make(): log the messages about making included Makefiles at log_level 4 instead of 3, so that verbosity 3 can be used to see the vars being used by the filtering process (i.e. the filtering process must still be enabled at verbosity 3)

9738 06/06/2013 02:00 AM Aaron Marcuse-Kubitza

bin/make: use `readlink -f` on BASH_SOURCE0 so that this script can also be run via a symlink

9737 06/06/2013 01:57 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: no_PATH_recursion(): also remove $top_symlink_dir_abs

9736 06/06/2013 01:56 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added $top_symlink_dir_abs

9735 06/06/2013 01:53 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: $top_dir_orig: renamed to $top_symlink_dir since this reflects better what the dir actually represents, and when it differs from $top_dir

9734 06/06/2013 01:33 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: make(): fixed func name in usage

9733 06/06/2013 01:32 AM Aaron Marcuse-Kubitza

bin/make: don't print make cmd by default, so that only `make` output is printed at verbosity 1

9732 06/06/2013 01:31 AM Aaron Marcuse-Kubitza

bin/make: don't reinvoke make() if the make filter has already been set up, as indicated by $is_outermost (instead, invoke make directly using exec)

9731 06/06/2013 01:30 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added $is_outermost, which stores whether the util.sh env has not yet set up

9730 06/06/2013 01:28 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_redirs_cmd(): added $cmd_name_log_inc kw param to allow echoing the command name at a different log_level than the command output

9729 06/06/2013 12:22 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: make(): log the filter setup operations at log_level 2 so they do not clutter up normal output

9728 06/06/2013 12:13 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: filter_fd: alias-expand filter_cmd

9727 06/06/2013 12:08 AM Aaron Marcuse-Kubitza

added bin/make, which runs make, hiding verbose messages about making included Makefiles. this should be used in preference to plain make, to avoid excessive log messages that prevent the user from seeing the core commands that are being run.

9726 06/06/2013 12:05 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added no_PATH_recursion(), which allows running a system command of the same name as the script

9725 06/06/2013 12:04 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added PATH_rm(), which removes components from the PATH

9724 06/06/2013 12:03 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added $top_dir_abs, $top_dir_orig

9723 06/05/2013 11:44 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: command(): use `builtin command` instead of `exec` so that options like -p (reset PATH) work properly. also, the command builtin it overrides is designed to be used with more than just external commands, and command() should not impose additional limitations.

9722 06/05/2013 10:54 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added self_sys alias, which uses only system utilities (`command -p`) instead of the current PATH

9721 06/05/2013 10:37 PM Aaron Marcuse-Kubitza

lib/sh/make.sh: make(): at verbosity < 3, hide messages about making included Makefiles. this makes the make output much more readable when a Makefile contains an include statement, because there won't be a ton of log messages every time a Makefile is included. this filtering is so useful that it probably makes sense to run make for any of our Makefiles using `lib/runscripts/util.run make ...` instead of plain make. compare e.g. `make inputs/ACAD/Specimen/map.csv` (53 lines of output) and `lib/runscripts/util.run make inputs/ACAD/Specimen/map.csv` (17 lines of output, 1/3 as much).

9720 06/05/2013 10:15 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: log++: before cmd: documented that you need to use "log++" instad of log++ to avoid using the log++ alias, which prepends a log_local call. omitting the quotes is generally not a problem, but when there is another command wrapping the log++, you need the "" to avoid the wrapper applying to log_local's declare call instead.

9719 06/05/2013 10:09 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: log++: before cmd: documented how to use it as `log+ #` when incrementing multiple log_levels at once (this is a better method than `"log++" "log++" ...`)

9718 06/05/2013 10:04 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: log++: before cmd: documented that you need to use it as `"log++" "log++"` when repeating it multiple times

9717 06/05/2013 10:02 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added filter_fd(), which encapsulates the use of >() process substitution for filtering an fd other than stdout (yes, this is possible without lots of 3>&1 1>&2 2>&3 redirections!). this can be useful e.g. to filter logging output or highlight errors.

9716 06/05/2013 09:49 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: log+(): documented that with a cmd, assignments are applied just to the cmd, so log_local is not needed

9715 06/05/2013 08:59 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: moved pipe_delay() before fd-related functions so it can be used by them

9714 06/05/2013 08:56 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: removed no longer needed load new aliases before echo_stdin(), since pipe_delay() is now a function

9713 06/05/2013 08:48 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: set_fds(): add #<>&- before every #<>&# reopen: need to use loop var $i instead of $1 (which would have been used with a while/shift method of iterating over $@)

9712 06/05/2013 12:57 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: set_fds(): added workaround for strange bash bug where reopening an fd sometimes first requires explicitly closing it, by adding an <>&- entry for every redirection

9711 06/05/2013 12:53 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added match_prefix()

9710 06/05/2013 03:28 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: make(): determine --silent status based on the verbosity (<=0) instead of a kw param

9709 06/05/2013 03:11 AM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv.md5/make(): only use %-remake if remaking

9708 06/05/2013 03:10 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: make(): removed extra space after --silent

9707 06/04/2013 11:17 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: moved mysql_root() after the mysql->mysql_ANSI alias (and load new aliases) so that it will also use ANSI mode and support "" identifiers

9706 06/04/2013 11:15 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql: always use ANSI mode, to support "" identifiers. note that `` are still supported in this mode, so it also works with SHOW CREATE TABLE output and dumpfiles.

9705 06/04/2013 11:05 PM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/raw_occurrence_record/run: plant_fraction_for_herbaria_filter/make(): need to make prerequisites first (plant_fraction/make)

9704 06/04/2013 06:35 PM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/table.run: table.tsv.md5/make(): use %-remake to ensure that the .md5 file is remade, regardless of the .md5 file's mtime relative to table.tsv. you would generally expect table.tsv's new mtime to be newer than the .md5 file's (thus triggering make to run), but if you e.g. ran svn up after making the table.tsv, this might not be the case.

9703 06/04/2013 06:31 PM Aaron Marcuse-Kubitza

/Makefile: moved %/remake, %-remake to lib/common.Makefile because they are generally useful

9702 06/04/2013 06:26 PM Aaron Marcuse-Kubitza

/Makefile: moved %/reinstall to lib/common.Makefile because it is generally useful

9701 06/04/2013 06:07 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql_cmd(): run the command with `time` because in mysql()'s output_data mode, no queries, and therefore no runtimes, are echoed, so the total runtime needs to be echoed separately instead. the total runtime is also useful in general, when many long-running queries are run and you would also like to know the total time (e.g. in make_analytical_db).

9700 06/04/2013 05:54 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: limit(): usage: surrounded query in "" to clarify that it's a string, not a command

9699 06/01/2013 09:32 PM Aaron Marcuse-Kubitza

inputs/GBIF/table.run: table.tsv/make(): use new set_large_table to prevent table.tsv from being deleted on error for full export runs (while still deleting it on error for sample subset runs)

9698 06/01/2013 09:31 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: added set_large_table alias, used to set to_file's $del flag based on $limit