Project

General

Profile

Activity

From 05/10/2013 to 06/08/2013

06/08/2013

12:23 PM Revision 9779: inputs/GBIF/raw_occurrence_record_plants/run: added ih_herbarium/make(), which stores the IH herbaria
Aaron Marcuse-Kubitza
11:50 AM Revision 9778: bugfix: inputs/GBIF/raw_occurrence_record_plants/run: table/make(): also filter out rows with a non-plant family (as described at http://vegpath.org/wiki/2013-06-06_conference_call#GBIF-subsetting-fix-raw_occurrence_record-filter-formula), since some institutions have both animal and plant rows, even though they are in IH or in the 80% list. (note that NULL families are OK.)
Aaron Marcuse-Kubitza
04:12 AM Revision 9777: *{.sh,run}: use mysql instead of mysql_ANSI because mysql is now an alias to mysql_ANSI (since ANSI mode still supports key MySQL features, like `` quotes)
Aaron Marcuse-Kubitza
04:09 AM Revision 9776: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): documented that incremental output is provided right away with --quick (unbuffered), but takes awhile to become visible in Macfusion sshfs. this can be tested with `while true; do stat inputs/GBIF/raw_occurrence_record_plants/table.tsv; sleep 2; done` running concurrently with `./inputs/GBIF/raw_occurrence_record_plants/run table.tsv/make` on vegbiendev:/home/bien/svn .
Aaron Marcuse-Kubitza
04:00 AM Revision 9775: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): use new raw_occurrence_record_plants view from table/make()
Aaron Marcuse-Kubitza
03:15 AM Revision 9774: bugfix: inputs/GBIF/raw_occurrence_record_plants/run: table/make(): added make of prerequisites
Aaron Marcuse-Kubitza
03:14 AM Revision 9773: bugfix: inputs/GBIF/raw_occurrence_record_plants/run: table/make(): don't reset $table to plant_fraction_for_herbaria_filter for commands that use $table
Aaron Marcuse-Kubitza
03:10 AM Revision 9772: inputs/GBIF/raw_occurrence_record_plants/run: added table/make(), which makes the filter view
Aaron Marcuse-Kubitza
02:14 AM Revision 9771: inputs/GBIF/raw_occurrence_record/: renamed to raw_occurrence_record_plants because it's actually only the plants in raw_occurrence_record, not all of raw_occurrence_record. also, this will allow us to create a separate raw_occurrence_record_plants view whose name matches the folder and does not collide with the raw_occurrence_record table.
Aaron Marcuse-Kubitza
12:44 AM Revision 9770: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): added runtime, which is ~0 since it just needs to do CSV import and index scans
Aaron Marcuse-Kubitza
12:43 AM Revision 9769: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): time the population of herbaria_filter
Aaron Marcuse-Kubitza

06/07/2013

11:47 PM Revision 9768: inputs/GBIF/raw_occurrence_record/run: plant_fraction/make(): updated runtime. added rows affected count to runtime so if the number of rows it's related to (in this case, institution_code) changes, the runtime can be expected to change accordingly.
Aaron Marcuse-Kubitza

06/06/2013

04:54 PM Revision 9767: inputs/.TNRS/schema.sql: tnrs_populate_fields(): documented runtime (17 min)
Aaron Marcuse-Kubitza
04:49 PM Revision 9766: bugfix: inputs/GBIF/raw_occurrence_record/run: plant_fraction/make(): plant_fraction column: COUNT(boolean) counts *non-NULL* rather than true values (which counter-intuitively includes false, because it's non-NULL), so need to add NULLIF(..., false) around the boolean expression to turn it into a NULL-or-not expression. see http://vegpath.org/wiki/2013-06-06_conference_call#GBIF-subsetting-fix-plant_fraction-SQL-bug .
Aaron Marcuse-Kubitza
04:29 PM Revision 9765: inputs/.TNRS/schema.sql: tnrs_populate_fields(): documented that when changing this function, you must regenerate the derived cols using `UPDATE tnrs SET "Name_submitted" = "Name_submitted"`
Aaron Marcuse-Kubitza
04:21 PM Revision 9764: inputs/.TNRS/schema.sql: tnrs_populate_fields(): Is_plant: must match family as Family_score = 1 (as discussed during conference call vegpath.org/wiki/2013-05-30_conference_call#postprocess-TNRS-results-to-exclude-animals-with-genus-homonyms) instead of as Family_matched IS NOT NULL (as listed in Brad's formula at vegpath.org/wiki/Result_filtering#TNRS-results) because TNRS transforms animal to plant families via fuzzy matching, necessitating a Family_score check to ensure an exact match to a plant family that was not transformed from an animal family
Aaron Marcuse-Kubitza
03:29 PM Revision 9763: inputs/.TNRS/schema.sql: added Is_plant derived field, which is populated using the formula at vegpath.org/wiki/Result_filtering#TNRS-results . note that the homonym filtering is currently excluded until we determine whether we can get direct access to the IRMNG homonyms database (http://www.cmar.csiro.au/datacentre/irmng/homonyms.htm). note also that changes to the TNRS schema cannot be fully tested until any TNRS client bugs are fixed, because the data.sql updater requires a working TNRS client to regenerate the sample data.
Aaron Marcuse-Kubitza
02:42 PM Revision 9762: inputs/.TNRS/schema.sql: updated for current TSV schema: renamed Accepted_species->Accepted_name_species, Accepted_family->Accepted_name_family
Aaron Marcuse-Kubitza
02:27 PM Revision 9761: bugfix: schemas/vegbien.sql: tnrs_input_name: must anti-join against MatchedTaxon rather than ValidMatchedTaxon to ensure that *all* of TNRS.tnrs is excluded from the input names. this prevents duplicates from appearing in the TNRS results, which would break the TSV import into TNRS.tnrs. it also prevents no-match names from being scrubbed repeatedly because they were not properly filtered out of the input names.
Aaron Marcuse-Kubitza
02:17 PM Revision 9760: inputs/.TNRS/schema.sql: fixed whitespace
Aaron Marcuse-Kubitza
02:15 PM Revision 9759: inputs/.TNRS/schema.sql: added MatchedTaxon view, which now just renames the columns but does not filter the results, and use it in ValidMatchedTaxon
Aaron Marcuse-Kubitza
02:11 PM Revision 9758: inputs/.TNRS/schema.sql: MatchedTaxon: renamed to ValidMatchedTaxon since this view actually contains only the names with a valid match
Aaron Marcuse-Kubitza
12:59 PM Revision 9757: bugfix: lib/sql.py: parse_exception(): make_DuplicateKeyException(): handle nested exceptions (which should never be generated, but may be in case of sql.py bugs such as wiki.vegpath.org/To_Do#Fixes > #1) by printing the nested exception and then rethrowing the original exception, so that the original exception does not get lost and still ends up at the end of the program's output, to enable debugging
Aaron Marcuse-Kubitza
11:08 AM Revision 9756: inputs/.TNRS/schema.sql: tnrs: documented that when changing this table's schema, you must regenerate data.sql using `inputs/test_taxonomic_names/test_scrub`
Aaron Marcuse-Kubitza
08:09 AM Revision 9755: inputs/GBIF/raw_occurrence_record/run: table.tsv.gz/make(): documented runtime (35 min)
Aaron Marcuse-Kubitza
07:52 AM Revision 9754: bugfix: schemas/vegbien.sql: analytical_stem_view: speciesBinomialWithMorphospecies: if accepted name not specified, use matched name (matched*) or Name_submitted (concatenatedScientificName), as described at http://wiki.vegpath.org/2013-05-30_conference_call#fix-TNRS-speciesBinomialWithMorphospecies-to-include-alternatives-when-no-accepted-name
Aaron Marcuse-Kubitza
07:24 AM Revision 9753: lib/sh/make.sh: make(): time all invocations of make
Aaron Marcuse-Kubitza
07:14 AM Revision 9752: lib/sh/make.sh: make() at verbosity < 4, hide messages about making included Makefiles: use sed with a range expression (/.../,/.../) to also exclude all log messages between an opening "make ...Makefile" and a closing "make[#]: ...Makefile"
Aaron Marcuse-Kubitza
06:35 AM Revision 9751: lib/sh/util.sh: log+ aliases: added clog++/-- aliases for cmds, which don't include log_local. these are useful when you can't just use "log++" because you need the command following it to be alias-expanded.
Aaron Marcuse-Kubitza
05:52 AM Revision 9750: bugfix: lib/sh/make.sh: make(): use [:char_class:] exprs instead of \X char class abbrs because the \ abbrs are not supported on Linux
Aaron Marcuse-Kubitza
05:33 AM Revision 9749: inputs/GBIF/table.run: table.tsv/make(): remake table.tsv.gz/make() after table.tsv is made
Aaron Marcuse-Kubitza
05:27 AM Revision 9748: inputs/GBIF/table.run: added table.tsv.gz/make()
Aaron Marcuse-Kubitza
05:26 AM Revision 9747: lib/sh/util.sh: added bg_cmd() and helpers log_bg(), log_last_bg()
Aaron Marcuse-Kubitza
05:04 AM Revision 9746: lib/runscripts/subdir.run: auto-append -remake to all targets in remake mode
Aaron Marcuse-Kubitza
04:15 AM Revision 9745: lib/runscripts/subdir.run: subdir_make(): put prepending of "$subdir/" on its own line for clarity
Aaron Marcuse-Kubitza
04:00 AM Revision 9744: bugfix: lib/common.Makefile: gzip/gunzip: in `touch -r`, *$<* needs to be the reference file
Aaron Marcuse-Kubitza
02:36 AM Revision 9743: *{.sh,run}: make() calls: removed no longer applicable silent=1, which is now handled automatically by the log_level mechanism
Aaron Marcuse-Kubitza
02:35 AM Revision 9742: /Makefile: new config target (part of install): install our bin/make in the user's ~/bin dir so that bin/make (with filtering of verbose messages) will always be used instead of standard make (without us needing to change every occurrence of make to bin/make!)
Aaron Marcuse-Kubitza
02:25 AM Revision 9741: bugfix: bin/make: use standard make logging port (1) instead of $log_fd (30) so that the output of bin/make can in turn be filtered by util.sh using the standard cmd_log_fd=1
Aaron Marcuse-Kubitza
02:23 AM Revision 9740: lib/sh/util.sh: command(): print the log_fd in effect in case the user changes it
Aaron Marcuse-Kubitza
02:23 AM Revision 9739: lib/sh/make.sh: make(): log the messages about making included Makefiles at log_level 4 instead of 3, so that verbosity 3 can be used to see the vars being used by the filtering process (i.e. the filtering process must still be enabled at verbosity 3)
Aaron Marcuse-Kubitza
02:00 AM Revision 9738: bin/make: use `readlink -f` on BASH_SOURCE[0] so that this script can also be run via a symlink
Aaron Marcuse-Kubitza
01:57 AM Revision 9737: lib/sh/util.sh: no_PATH_recursion(): also remove $top_symlink_dir_abs
Aaron Marcuse-Kubitza
01:56 AM Revision 9736: lib/sh/util.sh: added $top_symlink_dir_abs
Aaron Marcuse-Kubitza
01:53 AM Revision 9735: lib/sh/util.sh: $top_dir_orig: renamed to $top_symlink_dir since this reflects better what the dir actually represents, and when it differs from $top_dir
Aaron Marcuse-Kubitza
01:33 AM Revision 9734: lib/sh/make.sh: make(): fixed func name in usage
Aaron Marcuse-Kubitza
01:32 AM Revision 9733: bin/make: don't print make cmd by default, so that only `make` output is printed at verbosity 1
Aaron Marcuse-Kubitza
01:31 AM Revision 9732: bin/make: don't reinvoke make() if the make filter has already been set up, as indicated by $is_outermost (instead, invoke make directly using exec)
Aaron Marcuse-Kubitza
01:30 AM Revision 9731: lib/sh/util.sh: added $is_outermost, which stores whether the util.sh env has not yet set up
Aaron Marcuse-Kubitza
01:28 AM Revision 9730: lib/sh/util.sh: echo_redirs_cmd(): added $cmd_name_log_inc kw param to allow echoing the command name at a different log_level than the command output
Aaron Marcuse-Kubitza
12:22 AM Revision 9729: lib/sh/make.sh: make(): log the filter setup operations at log_level 2 so they do not clutter up normal output
Aaron Marcuse-Kubitza
12:13 AM Revision 9728: lib/sh/util.sh: filter_fd: alias-expand filter_cmd
Aaron Marcuse-Kubitza
12:08 AM Revision 9727: added bin/make, which runs make, hiding verbose messages about making included Makefiles. this should be used in preference to plain make, to avoid excessive log messages that prevent the user from seeing the core commands that are being run.
Aaron Marcuse-Kubitza
12:05 AM Revision 9726: lib/sh/util.sh: added no_PATH_recursion(), which allows running a system command of the same name as the script
Aaron Marcuse-Kubitza
12:04 AM Revision 9725: lib/sh/util.sh: added PATH_rm(), which removes components from the PATH
Aaron Marcuse-Kubitza
12:03 AM Revision 9724: lib/sh/util.sh: added $top_dir_abs, $top_dir_orig
Aaron Marcuse-Kubitza

06/05/2013

11:44 PM Revision 9723: bugfix: lib/sh/util.sh: command(): use `builtin command` instead of `exec` so that options like -p (reset PATH) work properly. also, the command builtin it overrides is designed to be used with more than just external commands, and command() should not impose additional limitations.
Aaron Marcuse-Kubitza
10:54 PM Revision 9722: lib/sh/util.sh: added self_sys alias, which uses only system utilities (`command -p`) instead of the current PATH
Aaron Marcuse-Kubitza
10:37 PM Revision 9721: lib/sh/make.sh: make(): at verbosity < 3, hide messages about making included Makefiles. this makes the make output **much** more readable when a Makefile contains an include statement, because there won't be a ton of log messages every time a Makefile is included. this filtering is so useful that it probably makes sense to run make for any of our Makefiles using `lib/runscripts/util.run make ...` instead of plain make. compare e.g. `make inputs/ACAD/Specimen/map.csv` (53 lines of output) and `lib/runscripts/util.run make inputs/ACAD/Specimen/map.csv` (17 lines of output, 1/3 as much).
Aaron Marcuse-Kubitza
10:15 PM Revision 9720: lib/sh/util.sh: log++: before cmd: documented that you need to use "log++" instad of log++ to avoid using the log++ alias, which prepends a log_local call. omitting the quotes is generally not a problem, but when there is another command wrapping the log++, you need the "" to avoid the wrapper applying to log_local's declare call instead.
Aaron Marcuse-Kubitza
10:09 PM Revision 9719: lib/sh/util.sh: log++: before cmd: documented how to use it as `log+ #` when incrementing multiple log_levels at once (this is a better method than `"log++" "log++" ...`)
Aaron Marcuse-Kubitza
10:04 PM Revision 9718: lib/sh/util.sh: log++: before cmd: documented that you need to use it as `"log++" "log++"` when repeating it multiple times
Aaron Marcuse-Kubitza
10:02 PM Revision 9717: lib/sh/util.sh: added filter_fd(), which encapsulates the use of >() process substitution for filtering an fd other than stdout (yes, this is possible without lots of 3>&1 1>&2 2>&3 redirections!). this can be useful e.g. to filter logging output or highlight errors.
Aaron Marcuse-Kubitza
09:49 PM Revision 9716: lib/sh/util.sh: log+(): documented that with a cmd, assignments are applied just to the cmd, so log_local is not needed
Aaron Marcuse-Kubitza
08:59 PM Revision 9715: lib/sh/util.sh: moved pipe_delay() before fd-related functions so it can be used by them
Aaron Marcuse-Kubitza
08:56 PM Revision 9714: lib/sh/util.sh: removed no longer needed load new aliases before echo_stdin(), since pipe_delay() is now a function
Aaron Marcuse-Kubitza
08:48 PM Revision 9713: bugfix: lib/sh/util.sh: set_fds(): add #<>&- before every #<>&# reopen: need to use loop var $i instead of $1 (which would have been used with a while/shift method of iterating over $@)
Aaron Marcuse-Kubitza
12:57 PM Revision 9712: bugfix: lib/sh/util.sh: set_fds(): added workaround for strange bash bug where reopening an fd sometimes first requires explicitly closing it, by adding an <>&- entry for every redirection
Aaron Marcuse-Kubitza
12:53 PM Revision 9711: lib/sh/util.sh: added match_prefix()
Aaron Marcuse-Kubitza
03:28 AM Revision 9710: lib/sh/make.sh: make(): determine --silent status based on the verbosity (<=0) instead of a kw param
Aaron Marcuse-Kubitza
03:11 AM Revision 9709: inputs/GBIF/table.run: table.tsv.md5/make(): only use %-remake if remaking
Aaron Marcuse-Kubitza
03:10 AM Revision 9708: lib/sh/make.sh: make(): removed extra space after --silent
Aaron Marcuse-Kubitza

06/04/2013

11:17 PM Revision 9707: lib/sh/db.sh: moved mysql_root() after the mysql->mysql_ANSI alias (and load new aliases) so that it will also use ANSI mode and support "" identifiers
Aaron Marcuse-Kubitza
11:15 PM Revision 9706: lib/sh/db.sh: mysql: always use ANSI mode, to support "" identifiers. note that `` are *still* supported in this mode, so it also works with SHOW CREATE TABLE output and dumpfiles.
Aaron Marcuse-Kubitza
11:05 PM Revision 9705: bugfix: inputs/GBIF/raw_occurrence_record/run: plant_fraction_for_herbaria_filter/make(): need to make prerequisites first (plant_fraction/make)
Aaron Marcuse-Kubitza
06:35 PM Revision 9704: bugfix: inputs/GBIF/table.run: table.tsv.md5/make(): use %-remake to ensure that the .md5 file is remade, regardless of the .md5 file's mtime relative to table.tsv. you would generally expect table.tsv's new mtime to be newer than the .md5 file's (thus triggering make to run), but if you e.g. ran svn up after making the table.tsv, this might not be the case.
Aaron Marcuse-Kubitza
06:31 PM Revision 9703: /Makefile: moved %/remake, %-remake to lib/common.Makefile because they are generally useful
Aaron Marcuse-Kubitza
06:26 PM Revision 9702: /Makefile: moved %/reinstall to lib/common.Makefile because it is generally useful
Aaron Marcuse-Kubitza
06:07 PM Revision 9701: lib/sh/db.sh: mysql_cmd(): run the command with `time` because in mysql()'s output_data mode, no queries, and therefore no runtimes, are echoed, so the total runtime needs to be echoed separately instead. the total runtime is also useful in general, when many long-running queries are run and you would also like to know the total time (e.g. in make_analytical_db).
Aaron Marcuse-Kubitza
05:54 PM Revision 9700: lib/sh/db.sh: limit(): usage: surrounded query in "" to clarify that it's a string, not a command
Aaron Marcuse-Kubitza

06/01/2013

09:32 PM Revision 9699: inputs/GBIF/table.run: table.tsv/make(): use new set_large_table to prevent table.tsv from being deleted on error for full export runs (while still deleting it on error for sample subset runs)
Aaron Marcuse-Kubitza
09:31 PM Revision 9698: lib/sh/db.sh: added set_large_table alias, used to set to_file's $del flag based on $limit
Aaron Marcuse-Kubitza
09:30 PM Revision 9697: lib/sh/util.sh: added exit2bool()
Aaron Marcuse-Kubitza
09:18 PM Revision 9696: lib/sh/util.sh: int2bool(): renamed to int2exit() because it actually sets a boolean exit status rather than returning a boolean string
Aaron Marcuse-Kubitza
09:07 PM Revision 9695: bugfix: lib/sh/util.sh: to_file(): also kw_params the other kw params if_not_exists, del
Aaron Marcuse-Kubitza
07:02 PM Revision 9694: lib/sh/db.sh: mysql_cmd(): use --quick to avoid buffering entire result (which is slow and memory-intensive for large result sets). this option applies to both mysql() and mysqldump().
Aaron Marcuse-Kubitza
06:35 PM Revision 9693: lib/sh/make.sh: self_make(): documented that it should be preceded by set_make_vars
Aaron Marcuse-Kubitza
06:34 PM Revision 9692: bugfix: inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.run: ^.preamble.sql/make(): need to run set_make_vars even though the make vars are not used, because set_make_vars sets $_remake, which is needed by self_make
Aaron Marcuse-Kubitza
06:33 PM Revision 9691: lib/sh/make.sh: set_make_vars: echo_vars $_remake to help debugging
Aaron Marcuse-Kubitza
06:27 PM Revision 9690: bugfix: lib/sh/util.sh: to_file(): need to run invoked cmd using redir so that >$stdout redir is applied properly when cmd is a shell function instead of an external command (external commands were already redirected properly because command() calls redir). this fixes a bug in inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.run ^.preamble.sql/make(), where the generated file would be output to stdout instead of to the file because to_file()'s command was a shell function, and therefore the redirection was not applied. this fix requires redir() to be a separate function from command(), because command() does many things that are only applicable to external commands.
Aaron Marcuse-Kubitza
06:20 PM Revision 9689: lib/sh/util.sh: redir(): override save_e and add `unset redirs` so error handlers are not redirected
Aaron Marcuse-Kubitza
06:15 PM Revision 9688: lib/sh/util.sh: added alias_append(), similar to func_override() for aliases
Aaron Marcuse-Kubitza
06:05 PM Revision 9687: lib/sh/util.sh: redir(): don't do redir actions if redir will be run later (i.e. if command to run starts with `redir` or `command`)
Aaron Marcuse-Kubitza
06:04 PM Revision 9686: lib/sh/util.sh: redir(), command(): log their function calls at the usual log_level (2) instead of one higher, so that they are printed in the call tree to help debugging
Aaron Marcuse-Kubitza
05:39 PM Revision 9685: lib/sh/util.sh: added redir() and use it in command() to perform and echo the redirections
Aaron Marcuse-Kubitza
09:35 AM Revision 9684: lib/sh/util.sh: command(): removed comment that "the following redirections must happen in exactly this order", because there is now only one redirection
Aaron Marcuse-Kubitza
06:01 AM Revision 9683: lib/sh/util.sh: command(): removed 2>&$err_fd, which is no longer needed because $err_fd is now 2 (in general, there is probably not a need for a special $err_fd var, because 2 is already stderr)
Aaron Marcuse-Kubitza
05:52 AM Revision 9682: lib/sh/util.sh: command(): don't set err_fd to global stderr, because this prevents errors from being captured and parsed by callers. it is also not necessary to use a separate port than stderr, because stderr already contains only errors (since logging messages go to their own port). global stderr would still be useful e.g. for displaying input prompts the user when reading from global stdin.
Aaron Marcuse-Kubitza
05:17 AM Revision 9681: bugfix: *run: overriding targets: use new self_make to properly progagate the $remake flag to the overridden target, so that the target itself is not skipped
Aaron Marcuse-Kubitza
05:15 AM Revision 9680: lib/sh/util.sh: to_file(): added del= flag to prevent the file from being auto-removed on error (e.g. to preserve a partial result, which normally would be removed)
Aaron Marcuse-Kubitza
05:12 AM Revision 9679: lib/sh/make.sh: added self_make(), which progagates the $remake flag (normally it is not progagated, because prerequisites should not also be remade)
Aaron Marcuse-Kubitza
03:58 AM Revision 9678: lib/sh/util.sh: usage comments: when there is a descriptive comment on the same line as the usage, prepend it with # (as if it were an end-of-line comment) instead of enclosing it in (), to make it visually obvious that it's a comment and not part of the usage commands
Aaron Marcuse-Kubitza
03:54 AM Revision 9677: lib/sh/util.sh: added all_funcs(), which lists all declared functions
Aaron Marcuse-Kubitza
03:31 AM Revision 9676: bugfix: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added check_target_exists so table.tsv would not be overwritten if it already existed
Aaron Marcuse-Kubitza
03:30 AM Revision 9675: bugfix: inputs/GBIF/table.run: table.tsv/make(): force table.tsv.md5 to be remade (using remake=1) because the table.tsv contents will have changed
Aaron Marcuse-Kubitza
03:29 AM Revision 9674: bugfix: lib/sh/make.sh: set_make_vars: can't use end-of-line comment in alias because it will comment out whatever is after the alias where it's used. can't just put a newline or ; after the end-of-line comment because the alias's lines will be combined onto one line using ; , so end-of-line comments would not be supported.
Aaron Marcuse-Kubitza
03:23 AM Revision 9673: lib/sh/util.sh: added pf(), to print a function declaration for debugging
Aaron Marcuse-Kubitza
02:08 AM Revision 9672: inputs/GBIF/raw_occurrence_record/run: plant_fraction/make(): documented runtime (1 hr)
Aaron Marcuse-Kubitza
01:19 AM Revision 9671: *.url: mailto URLs: use the standard e-mail dotpath syntax e-mail@host?name.date.subject.(attachment) (vegpath.org/wiki/Global_IDs#Resource). populated missing fields (e.g. name, subject) where needed.
Aaron Marcuse-Kubitza
12:12 AM Revision 9670: *.url: mailto URLs: ensured that they are proper URLs with the mailto: protocol
Aaron Marcuse-Kubitza

05/30/2013

08:32 PM Revision 9669: lib/sh/util.sh: to_file(): exc handler that rm's file: unset redirs so it isn't used in the rm cmd
Aaron Marcuse-Kubitza
08:22 PM Revision 9668: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): removed no longer needed explicit clear of $remake, which is now done by make.sh instead
Aaron Marcuse-Kubitza
08:18 PM Revision 9667: lib/sh/make.sh: set_make_vars: don't progagate remake to prerequisites, so that remake=1 only applies to the outermost target rather than forcing every prerequisite to be remade, too
Aaron Marcuse-Kubitza
07:58 PM Revision 9666: lib/sh/make.sh: moved remaking section before set_make_vars so that it can be used in set_make_vars
Aaron Marcuse-Kubitza
07:53 PM Revision 9665: inputs/GBIF/raw_occurrence_record/run: added herbaria_filter/seal()
Aaron Marcuse-Kubitza
07:51 PM Revision 9664: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): changed "from IH" to "contains all of IH" because not all rows are now from IH
Aaron Marcuse-Kubitza
07:49 PM Revision 9663: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): renamed acronym->institution_code to match the column name in raw_occurrence_record rather than in IH
Aaron Marcuse-Kubitza
07:46 PM Revision 9662: inputs/GBIF/raw_occurrence_record/run: removed no longer used herbaria_filter.plant_fraction.csv_/make(). use plant_fraction_for_herbaria_filter view instead.
Aaron Marcuse-Kubitza
07:45 PM Revision 9661: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): use the plant_fraction_for_herbaria_filter view directly instead of first exporting it to a CSV
Aaron Marcuse-Kubitza
07:19 PM Revision 9660: lib/sh/db.sh: mysql_import(): in append mode, use LOAD DATA IGNORE to allow inserting duplicate rows
Aaron Marcuse-Kubitza
07:09 PM Revision 9659: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): if remaking, turn off remake mode after doing this target's rm operations, so that prerequisite targets are not also remade
Aaron Marcuse-Kubitza
06:56 PM Revision 9658: lib/sh/util.sh: to_file(): removed no longer needed separate logging of >$stdout, which is now done by command()
Aaron Marcuse-Kubitza
06:50 PM Revision 9657: lib/sh/util.sh: echo_redirs_cmd(): use $@ in a subshell instead of manipulating the @redirs array directly, because operations on $@ (e.g. $#, $1, shift) are much simpler than the corresponding array operations ( ${#redirs[@]}, ${redirs[0]}, redirs=("${redirs[@]:1}") )
Aaron Marcuse-Kubitza
06:42 PM Revision 9656: lib/sh/util.sh: echo_redirs_cmd(): log each file redir with a separate log() statement, so each line is indented
Aaron Marcuse-Kubitza
06:38 PM Revision 9655: lib/sh/util.sh: added echo_redirs_cmd and use it in command() to print cmd
Aaron Marcuse-Kubitza
06:32 PM Revision 9654: lib/sh/util.sh: command(): print <>file redirects before command, because they introduce it
Aaron Marcuse-Kubitza
06:31 PM Revision 9653: lib/sh/util.sh: added starts_with()
Aaron Marcuse-Kubitza
05:49 PM Revision 9652: lib/sh/util.sh: to_file(): use @redirs to echo and set >$stdout instead of setting it manually, which is possible now that the command() @redirs bug has been fixed
Aaron Marcuse-Kubitza
05:43 PM Revision 9651: bugfix: lib/sh/util.sh: convention of fds to use for command-specific alternate stdin/stdout/stderr: changed to 40/41/42 because 10/11/12 are used by eval (which is used by set_fds()). use of fd 10/11/12 will cause hard-to-find silent bugs because exec will not print an error when these are used. documented why not to use other series of fds for this purpose:
# do NOT use 1x, which are used by eval (which is used by set_fds())
# do NOT use 2x, which are used as global stdin/...
Aaron Marcuse-Kubitza
02:43 PM Revision 9650: lib/sh/local.sh: psql(): use new psql() from db.sh instead of psql_script_vegbien/psql_verbose_vegbien. this requires setting local_pg_database=vegbien to replace vegbien_dest used by psql_*_vegbien.
Aaron Marcuse-Kubitza
02:38 PM Revision 9649: lib/sh/db.sh: psql(): set $PG* connection env vars from our connection vars ($server, $user, etc.). use use_pg to import $database so it can be different from $database for MySQL
Aaron Marcuse-Kubitza
02:31 PM Revision 9648: lib/sh/db.sh: added use_pg alias
Aaron Marcuse-Kubitza
02:31 PM Revision 9647: bugfix: lib/sh/db.sh: psql(): added missing `--set ON_ERROR_STOP=1 --quiet` opts from psql_script_vegbien
Aaron Marcuse-Kubitza
02:12 PM Revision 9646: lib/sh/db.sh: added psql(), which replaces psql_script_vegbien and psql_verbose_vegbien for general connections. it also supports separate command and stdin files, to allow using `\copy from pstdin`, with pstdin pointing to a separate, EOF-terminated CSV file instead of inlined with the command and terminated with the \. escape (which may be contained within the CSV file itself).
Aaron Marcuse-Kubitza
01:05 PM Revision 9645: bugfix: lib/sh/local.sh: psql(): $file can't both be passed as a --file param and be prefixed with the necessary \set schema, etc. commands, so instead include $file when cat-ing stdin
Aaron Marcuse-Kubitza
08:28 AM Revision 9644: added inputs/GBIF/raw_occurrence_record/postprocess.sql, which removes institutions that we have direct data for
Aaron Marcuse-Kubitza
08:18 AM Revision 9643: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): skip table if already exists (unless remaking), like plant_fraction/make()
Aaron Marcuse-Kubitza
08:16 AM Revision 9642: bugfix: lib/sh/db.sh: mysql_import(): need to use direct connection to DB instead of via ssh, because ssh does not tunnel nonstandard fds
Aaron Marcuse-Kubitza
08:15 AM Revision 9641: lib/sh/db.sh: added ssh2local alias
Aaron Marcuse-Kubitza
07:36 AM Revision 9640: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.plant_fraction.csv_/make(): use new plant_fraction_for_herbaria_filter view
Aaron Marcuse-Kubitza
07:13 AM Revision 9639: inputs/GBIF/raw_occurrence_record/run: added plant_fraction_for_herbaria_filter/make(). note that for simplicity, plant_fraction_for_herbaria_filter is a view instead of a table.
Aaron Marcuse-Kubitza
06:50 AM Revision 9638: inputs/GBIF/raw_occurrence_record/run: *.table/*(): renamed to */*() because a target named after a table refers to the table unless it has an explicit file extension
Aaron Marcuse-Kubitza
06:49 AM Revision 9637: inputs/GBIF/raw_occurrence_record/run: plant_fraction.table/*(): renamed to plant_fraction/*() because a target named after a table refers to the table unless it has an explicit file extension
Aaron Marcuse-Kubitza
06:41 AM Revision 9636: lib/sh/db.sh: mysql_seal_table(): also revoke GRANT OPTION, which apparently needs to be done in addition (and in a separate command, unlike when granting GRANT OPTION)
Aaron Marcuse-Kubitza
06:40 AM Revision 9635: lib/sh/db.sh: mysql_seal_table(): REVOKE: ignore errors if REVOKE was already run
Aaron Marcuse-Kubitza
06:39 AM Revision 9634: lib/sh/db.sh: mysql_seal_table(): REVOKE: removed unneeded explicit database since this is automatically set to the current database
Aaron Marcuse-Kubitza
06:19 AM Revision 9633: inputs/GBIF/raw_occurrence_record/run: added plant_fraction.table/seal(), which uses new mysql_seal_table()
Aaron Marcuse-Kubitza
06:19 AM Revision 9632: lib/sh/db.sh: added mysql_seal_table(), which prevents further modifications to a table by a user. this uses new mysql_root().
Aaron Marcuse-Kubitza
06:18 AM Revision 9631: lib/sh/db.sh: added mysql_root(). this version uses just use_root (compare to the mysql_root() override in local.sh).
Aaron Marcuse-Kubitza
06:16 AM Revision 9630: lib/sh/local.sh: database connection vars: connect to vegbiendev via ssh and run commands locally, to allow running commands as root (which can only connect to the database locally). this effectively requires an ssh account on vegbiendev, but any ssh account (including an anonymous one, if we set one up) will do. this causes schemas/VegCore/VegCore.my.sql, VegCore.pg.sql to change, because they are now created by mysqldump running on vegbiendev (Linux) instead of on a Mac.
Aaron Marcuse-Kubitza

05/29/2013

10:35 PM Revision 9629: inputs/GBIF/raw_occurrence_record/run: plant_fraction: added index on plant_fraction for fast extraction of herbaria by fraction threshold
Aaron Marcuse-Kubitza
10:10 PM Revision 9628: inputs/GBIF/raw_occurrence_record/run: tables: set ENGINE to MyISAM and DEFAULT CHARSET to utf8 to match the other GBIF tables. (note that MyISAM is *not* the default, but is needed to avoid row sort order problems and other issues with InnoDB.)
Aaron Marcuse-Kubitza
08:09 PM Revision 9627: inputs/GBIF/raw_occurrence_record/run: plant_fraction.table/make(): in remaking mode, drop the table first
Aaron Marcuse-Kubitza
08:04 PM Revision 9626: inputs/GBIF/raw_occurrence_record/run: plant_fraction.table/make(): only create and populate the table if it doesn't already exist, to avoid clobbering existing data. the noclobber functionality uses new skip_table(), which is the table analog of require_not_exists().
Aaron Marcuse-Kubitza
08:02 PM Revision 9625: lib/runscripts/table.run, table.run: use new db_make.sh
Aaron Marcuse-Kubitza
08:02 PM Revision 9624: added lib/sh/db_make.sh that includes both db.sh and make.sh, and will eventually contain DB-related make commands
Aaron Marcuse-Kubitza
08:00 PM Revision 9623: lib/sh/db.sh: added skip_table(), which prints an already_exists_msg for tables
Aaron Marcuse-Kubitza
07:56 PM Revision 9622: lib/sh/util.sh: already_exists_msg: undid r9621 because the `|| return 0` should actually always be explicitly specified by the caller, to make it clear that the function will be aborted
Aaron Marcuse-Kubitza
07:47 PM Revision 9621: lib/sh/util.sh: already_exists_msg(): added alias for use as an error handler. note that ..._not_exists() functions should continue to use the "already_exists_msg" function instead to preserve the exit status.
Aaron Marcuse-Kubitza
07:40 PM Revision 9620: lib/sh/util.sh: added already_exists_msg() and use it instead of manually generating the die() call
Aaron Marcuse-Kubitza
07:15 PM Revision 9619: schemas/my.cnf: added innodb_file_per_table so each InnoDB table will get its own file. this should also allow databases with InnoDB tables to be manually renamed.
Aaron Marcuse-Kubitza
07:09 PM Revision 9618: added schemas/my.cnf from /etc/mysql/my.cnf
Aaron Marcuse-Kubitza
06:51 PM Revision 9617: schemas/VegCore/VegCore.my.sql, VegCore.pg.sql: synced to VegCore MySQL DB. for some reason, the fkeys are now output in the opposite order from what they were in before.
Aaron Marcuse-Kubitza
05:22 PM Revision 9616: inputs/.TNRS/schema.sql: MatchedTaxon: filter out rows where Max_score was not high enough to use the TNRS result as a match. removed now-duplicated filter for this in AcceptedTaxon.
Aaron Marcuse-Kubitza
05:19 PM Revision 9615: inputs/.TNRS/schema.sql: ScrubbedTaxon: removed extra ; at end of WHERE clause
Aaron Marcuse-Kubitza
03:48 PM Revision 9614: web/links/index.htm: updated to Firefox bookmarks. some broken favicons have also been fixed, by reopening bookmark in Firefox. (this will only update a favicon *if* there is a newer version. to delete a favicon completely, use Firefox's SQLite Manager plugin.)
Aaron Marcuse-Kubitza
10:17 AM Revision 9613: web/index.php: use XHTML DOCTYPE to match what's used by mod_autoindex. this requires some adjustments in spacing for XHTML's slightly different formatting
Aaron Marcuse-Kubitza
10:15 AM Revision 9612: bugfix: web/.htaccess: need to do DirectoryIndex redirects *before* checking for existing file/dir, because a DirectoryIndexed dir is existing but still needs to be redirected to the index.* file
Aaron Marcuse-Kubitza
10:01 AM Revision 9611: web/.htaccess: mod_autoindex: use the main.css stylesheet to match the look-and-feel of index.php
Aaron Marcuse-Kubitza
10:00 AM Revision 9610: web/.htaccess: mod_autoindex: Note that some listed files are not web-accessible: use ' instead of " to avoid \-escaping embedded "
Aaron Marcuse-Kubitza
09:39 AM Revision 9609: web/.htaccess: mod_autoindex: sort by description when provided, to allow setting a custom (non-alphabetical) sort order using AddDescription
Aaron Marcuse-Kubitza
09:37 AM Revision 9608: web/.htaccess: mod_autoindex: added note that some listed files are not web-accessible. they will produce a "Forbidden" error when clicked.
Aaron Marcuse-Kubitza
09:36 AM Revision 9607: bugfix: web/index.php: added space between the full directory index and the preceding content
Aaron Marcuse-Kubitza
09:35 AM Revision 9606: web/index.php: moved the full directory index within the rest of the document body
Aaron Marcuse-Kubitza
08:58 AM Revision 9605: web/index.php: include full directory index, since the URL patterns list is just a subset of the content available through vegpath.org
Aaron Marcuse-Kubitza
08:25 AM Revision 9604: web/.htaccess: added mod_autoindex IndexOptions, in particular FoldersFirst
Aaron Marcuse-Kubitza
05:26 AM Revision 9603: bugfix: web/.htaccess: changed "mod_dir listing"->"mod_autoindex listing" because mod_dir does not actually handle the autogenerated listings
Aaron Marcuse-Kubitza
05:24 AM Revision 9602: bugfix: web/.htaccess: DirectoryIndex: use disabled instead of on because on is actually treated as a filename, and does not invoke mod_autoindex. the DirectoryIndex directive and the mod_dir module actually apply only to manual index files, not to autogenerated dir listings (which are handled by mod_autoindex).
Aaron Marcuse-Kubitza
04:58 AM Revision 9601: web/index.php: removed no longer needed custom alias j.mp/vegpath# for when page reached through vegbiendev.nceas.ucsb.edu, because vegpath.org is a much more reliable domain than the previous path.vg, and a separate way to reach VegPath when path.vg is down is no longer needed
Aaron Marcuse-Kubitza
04:43 AM Revision 9600: web/.htaccess: <dir>/all forces mod_dir listing: use simpler $mod_dir_listing env var instead of query string modification to indicate that an explicit mod_dir listing should be displayed. this causes /all to replace ?index=1 as the way to force a mod_dir listing. note that the %{ENV:...} test needs to use $REDIRECT_mod_dir_listing instead of $mod_dir_listing, because a redirect will occur between the /all rule and the index.* rule, causing all env vars to be prepended with REDIRECT_ .
Aaron Marcuse-Kubitza
03:48 AM Revision 9599: web/.htaccess: <dir>/all forces mod_dir listing, as a simpler syntax than ?index=1
Aaron Marcuse-Kubitza
03:28 AM Revision 9598: web/.htaccess: for dirs, redirect to index.*: allow requesting a mod_dir listing instead with ?index=1
Aaron Marcuse-Kubitza
03:26 AM Revision 9597: web/.htaccess: handle DirectoryIndex redirects in a RewriteRule instead of with `DirectoryIndex index`, so that RewriteConds can be used to configure when index.* is used as the DirectoryIndex instead of a mod_dir listing
Aaron Marcuse-Kubitza
02:30 AM Revision 9596: web/.htaccess: handle DirectoryIndex subrequests when there is no DirectoryIndex: moved comment about -F subrequest after line it applies to
Aaron Marcuse-Kubitza
02:27 AM Revision 9595: inputs/GBIF/_MySQL/run: documented steps to reload GBIF MySQL
Aaron Marcuse-Kubitza
02:19 AM Revision 9594: web/.htaccess: RewriteRules: added standard [discardpath,noescape,qsappend] options where missing (these should be the default, but aren't)
Aaron Marcuse-Kubitza

05/24/2013

03:13 PM Revision 9593: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.table/make(): inline the PRIMARY KEY statement with its column
Aaron Marcuse-Kubitza
03:10 PM Revision 9592: bugfix: inputs/GBIF/raw_occurrence_record/run: plant_fraction.table/make(): create the table once with "IF NOT EXISTS" and then populate it with INSERT SELECT, to avoid locking it while it's being repopulated. dropping and recreating the table with CREATE TABLE AS prevented phpMyAdmin from even reading the database's tables list, because it was unable to fetch a rowcount for plant_fraction.
Aaron Marcuse-Kubitza
03:04 PM Revision 9591: lib/sh/db.sh: mysql(): when echoing queries, also echo runtimes (turned on with `--verbose --verbose --verbose`)
Aaron Marcuse-Kubitza
02:32 PM Revision 9590: added lib/runscripts/datasrc_dir.run
Aaron Marcuse-Kubitza
02:30 PM Revision 9589: inputs/GBIF/_MySQL/run: added load_data(), which loads the dumpfile into MySQL
Aaron Marcuse-Kubitza
02:06 PM Revision 9588: lib/sh/db.sh: added mysql_rm_privileged_statements()
Aaron Marcuse-Kubitza
02:00 PM Revision 9587: bugfix: lib/sh/resume_import.sh: sed calls: moved end-of-line comments to their own line because end-of-line comments are not supported on Mac
Aaron Marcuse-Kubitza
01:55 PM Revision 9586: lib/runscripts/table_dir.run: renamed table to subdir because this can apply to any datasrc subdir. moved table-specific code to table.run.
Aaron Marcuse-Kubitza
01:43 PM Revision 9585: lib/runscripts/table_dir.run: renamed table to subdir because this can apply to any datasrc subdir. moved table-specific code to table.run.
Aaron Marcuse-Kubitza
01:21 PM Revision 9584: lib/runscripts/table_dir.run: table_make(): moved $silent flag to lib/sh/make.sh make() so all make callers can use it
Aaron Marcuse-Kubitza
12:35 PM Revision 9583: bugfix: inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.run: override ^.preamble.sql/make() and use ../_src/GBIFPortalDB-2013-02-20.dump as the dumpfile instead of this file, which does not contain the preamble
Aaron Marcuse-Kubitza
12:23 PM Revision 9582: bugfix: lib/sh/resume_import.sh: $preamble_file: use the extension .0.preamble.sql instead of .preamble.sql so the preamble file sorts before the other *.sql files
Aaron Marcuse-Kubitza
12:22 PM Revision 9581: removed inputs/GBIF/_MySQL/MySQL.data.sql*, since we are using the much faster exported TSVs instead (see raw_occurrence_record/table.tsv). this also avoids confusion between GBIFPortalDB-2013-02-20.data.sql* and MySQL.data.sql* when loading data into MySQL.
Aaron Marcuse-Kubitza
12:18 PM Revision 9580: bugfix: inputs/GBIF/_MySQL/MySQL.data.sql.run: moved to GBIFPortalDB-2013-02-20.data.sql.run since it's actually the raw input file, not the ANSI export of it, that needs to be imported
Aaron Marcuse-Kubitza
12:16 PM Revision 9579: lib/sh/resume_import.sh: get_pkey_at_pos(): changed $quote to ` to work with inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql
Aaron Marcuse-Kubitza
11:50 AM Revision 9578: lib/sh/db.sh: mysql(): added $log_queries flag, which can be turned off to avoid using --verbose. this is useful when running bulk INSERT statements.
Aaron Marcuse-Kubitza
11:35 AM Revision 9577: lib/sh/local.sh: added mysql_local()
Aaron Marcuse-Kubitza
11:24 AM Revision 9576: lib/sh/local.sh: added mysql_root()
Aaron Marcuse-Kubitza
11:24 AM Revision 9575: lib/sh/local.sh: added $root_user, $root_password
Aaron Marcuse-Kubitza
11:22 AM Revision 9574: lib/sh/db.sh: added use_root alias (similar to use_local/use_remote)
Aaron Marcuse-Kubitza
11:21 AM Revision 9573: added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.schema.z.clean_up.sql, which removes duplicated and unnecessary indexes in raw_occurrence_record
Aaron Marcuse-Kubitza
11:20 AM Revision 9572: added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.schema.0.preamble.sql
Aaron Marcuse-Kubitza
11:02 AM Revision 9571: bugfix: lib/sh/resume_import.sh: sql_preamble(): also stop at first "-- Table structure for table" line (when using a full dumpfile rather than a data-only subset)
Aaron Marcuse-Kubitza
10:58 AM Revision 9570: lib/sh/resume_import.sh: resume_import(): run connection preamble (first few lines of dumpfile) before continuing with main file at offset, so that connection setting are reapplied
Aaron Marcuse-Kubitza
06:45 AM Revision 9569: lib/sh/resume_import.sh: is_pkey_imported__int(): use echo_stdout so the user can see the result of the > function in each iteration
Aaron Marcuse-Kubitza
06:42 AM Revision 9568: added lib/sh/resume_import.sh and use it in inputs/GBIF/_MySQL/MySQL.data.sql.run
Aaron Marcuse-Kubitza
06:32 AM Revision 9567: inputs/GBIF/_MySQL/MySQL.data.sql.run: is_pkey_imported__int(): made pkey name configurable in $pkey_name
Aaron Marcuse-Kubitza
05:32 AM Revision 9566: inputs/GBIF/_MySQL/MySQL.data.sql.run: import_resume_pos() run time: removed seconds because the precision is likely only to the nearest half-minute
Aaron Marcuse-Kubitza
05:31 AM Revision 9565: inputs/GBIF/_MySQL/MySQL.data.sql.run: documented that import_resume_pos() takes 6 min to run, with 37 iterations
Aaron Marcuse-Kubitza
05:20 AM Revision 9564: added inputs/GBIF/_MySQL/MySQL.data.sql.run, with helper functions for resuming the import to MySQL from where it left off. this is very useful if the import is interrupted for any reason, because otherwise, the entire import would have to be run again from the start, taking 40-50 hours. import_resume_pos() uses new binsearch() to find where in the file the import left off, based on which pkeys have already been imported. (GBIF pkeys are unfortnately not in any order in the input file, nor are they in insertion order in the imported table, because MySQL instead clusters the table by the pkey. this necessitates a much more complex solution to resuming a partial import.)
Aaron Marcuse-Kubitza
05:14 AM Revision 9563: lib/sh/binsearch.sh: binsearch(): also echo_vars the iter_num, to track how close binsearch is to finding the value (it will always take the same # iters, log2(max - min) )
Aaron Marcuse-Kubitza
05:11 AM Revision 9562: lib/sh/binsearch.sh: binsearch(): also echo_vars the min/max so these can be used as shortcut inputs if binsearch is run again
Aaron Marcuse-Kubitza
04:58 AM Revision 9561: bugfix: lib/sh/util.sh: caching: cache_key for function inputs: need to use `declare -p kw_param` instead of "$kw_param" because declare accepts a param name, not value`
Aaron Marcuse-Kubitza
03:40 AM Revision 9560: lib/sh/binsearch.sh: binsearch(): doc comment: fixed typo in "truncates"
Aaron Marcuse-Kubitza
03:17 AM Revision 9559: bugfix: lib/sh/util.sh: func_override(): need to match *shortest* __* suffix instead of longest in case the function being overridden itself contained __
Aaron Marcuse-Kubitza
01:51 AM Revision 9558: bugfix: lib/sh/util.sh: file_size: Linux: need % in %s
Aaron Marcuse-Kubitza
01:43 AM Revision 9557: lib/sh/db.sh: mysql(): added $data_only flag which enables --skip-column-names and $output_data
Aaron Marcuse-Kubitza
01:41 AM Revision 9556: bugfix: lib/sh/util.sh: file_size: need to use --format instead of -f on Linux
Aaron Marcuse-Kubitza
01:22 AM Revision 9555: added lib/runscripts/table_dir.run and use it in table.run
Aaron Marcuse-Kubitza
01:20 AM Revision 9554: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.ih.csv_/make(): don't use any outer limit value, so that all the IH herbaria are always used. this also ensures that the first GBIF rows will be from an IH herbarium.
Aaron Marcuse-Kubitza
01:17 AM Revision 9553: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.table/make(): herbaria_filter: don't explicitly set ENGINE or DEFAULT CHARSET, because these should be set to the database values instead so that collations, etc. match
Aaron Marcuse-Kubitza
12:50 AM Revision 9552: lib/sh/util.sh: filesystem: added file_size alias
Aaron Marcuse-Kubitza
12:34 AM Revision 9551: lib/sh/util.sh: exceptions: added signals-related functions ignore_sig(), piped_cmd() and helper sig_e()
Aaron Marcuse-Kubitza

05/23/2013

11:40 PM Revision 9550: lib/sh/util.sh: $sed_cmd: don't use `command`, which causes sed calls (which are usually internal) to always be logged. instead, use echo_run wherever sed needs to be logged.
Aaron Marcuse-Kubitza
11:38 PM Revision 9549: lib/sh/util.sh: echo_run(): added trailing-space alias to alias-expand next word, which is a command
Aaron Marcuse-Kubitza
11:31 PM Revision 9548: lib/sh/binsearch.sh: binsearch(): echo $i at log_level 1 so it's displayed by default, as a progress indicator
Aaron Marcuse-Kubitza
11:30 PM Revision 9547: lib/sh/binsearch.sh: binsearch(): echo $i at log_level 1 so it's displayed by default, as a progress indicator
Aaron Marcuse-Kubitza
11:29 PM Revision 9546: lib/sh/binsearch.sh: binsearch(): echo the command being run using new echo_run()
Aaron Marcuse-Kubitza
11:25 PM Revision 9545: lib/sh/util.sh: log+: set PS4 from $log_level instead of relative to its previous value. this allows PS4 to work properly at negative log_levels, in spite of the inability to store a "negative" value in a prefix string.
Aaron Marcuse-Kubitza
11:23 PM Revision 9544: lib/sh/util.sh: added float_set_min()
Aaron Marcuse-Kubitza
11:22 PM Revision 9543: lib/sh/util.sh: log+(): log_level: set it using simpler $(()), since log_level will never be fractional (although verbosity can be). log_level may of course be fractional in invoked scripts, but that does not affect util.sh.
Aaron Marcuse-Kubitza
10:44 PM Revision 9542: lib/sh/util.sh: log++: also track a numeric log_level var, which follows the PS4 prefix
Aaron Marcuse-Kubitza
10:35 PM Revision 9541: inputs/.TNRS/schema.sql: MatchedTaxon: matchedFamily: use Accepted_family when the Name_matched_accepted_family is not provided, as it's omitted by the current TNRS CSV schema
Aaron Marcuse-Kubitza
09:54 PM Revision 9540: lib/sh/util.sh: log+(): PS4: split if statement onto multiple lines for clarity
Aaron Marcuse-Kubitza
09:44 PM Revision 9539: lib/sh/util.sh: added back echo_run(), usable for internal commands where command() would be used for external commands
Aaron Marcuse-Kubitza
09:33 PM Revision 9538: lib/sh/util.sh: added int2bool()
Aaron Marcuse-Kubitza
09:25 PM Revision 9537: *{.sh,run}: use new `|| ignore` instead of ignore_e/end_try
Aaron Marcuse-Kubitza
09:25 PM Revision 9536: lib/sh/util.sh: added ignore(), which uses ||-syntax
Aaron Marcuse-Kubitza
09:13 PM Revision 9535: lib/sh/util.sh: ignore(): renamed to ignore_e() so ignore() can be used for a simpler, ||-based command
Aaron Marcuse-Kubitza
09:09 PM Revision 9534: bugfix: lib/sh/util.sh: catch(): need && between test and e=0 so e=0 is only run if $e was equal to the desired value
Aaron Marcuse-Kubitza
08:22 PM Revision 9533: added lib/sh/binsearch.sh
Aaron Marcuse-Kubitza
06:27 PM Revision 9532: bugfix: README.TXT: Full database import: screen: need to unset TMOUT, version *after* running `screen` rather than before so they take effect within the `screen` shell
Aaron Marcuse-Kubitza
06:25 PM Revision 9531: README.TXT: Full database import: after running `screen`: run `set -o ignoreeof` to prevent Ctrl+D from exiting `screen` to keep attached jobs
Aaron Marcuse-Kubitza
04:40 PM Revision 9530: bin/tnrs_db: documented how to estimate total runtime. note that our tnrs_db wrapper in inputs/.TNRS/tnrs/tnrs.make uses inputs/.TNRS/tnrs/logs/tnrs.make.log.sql as the log file.
Aaron Marcuse-Kubitza
03:33 PM Revision 9529: inputs/.TNRS/schema.sql, data.sql: updated TNRS CSV columns to preserve Name_matched_accepted_family even though it isn't present in the current TNRS CSVs. this way, Name_matched_accepted_family can still be used for previously-scrubbed names, and family_matched can be added back to analytical_stem_view. (now that bin/tnrs_db uses an explicit columns list in COPY TO, the absence of a column in the CSV is no longer a problem.)
Aaron Marcuse-Kubitza
03:28 PM Revision 9528: README.TXT: updating TNRS CSV columns: use the entire "COPY tnrs ..." statement instead of just the body of it so that the explicit columns list is included. this way, the COPY statement will cause an error if the TNRS schema was changed but inputs/.TNRS/data.sql was not yet updated.
Aaron Marcuse-Kubitza
03:00 PM Revision 9527: bin/tnrs_db: removed unused imports
Aaron Marcuse-Kubitza
02:55 PM Revision 9526: bin/tnrs_db: cumulative_tnrs_profiler: use tnrs.tnrs_request()'s new cumulative_profiler param instead of doing the profiling manually. this also ensures that there isn't extra time between when the cumulative profiler starts/stops and when the per-request profiler starts/stops (because Profiler's new add_subprofiler() method is used).
Aaron Marcuse-Kubitza
02:53 PM Revision 9525: lib/tnrs.py: single_tnrs_request(): added support for a cumulative profiler using the cumulative_profiler kw param
Aaron Marcuse-Kubitza
02:53 PM Revision 9524: lib/profiling.py: Profiler: added add_subprofiler(), for use with cumulative profilers
Aaron Marcuse-Kubitza
02:48 PM Revision 9523: lib/profiling.py: Profiler: added add_time() and use it instead of `self.total +=`
Aaron Marcuse-Kubitza
02:38 PM Revision 9522: bin/tnrs_db: tnrs_profiler: renamed to cumulative_tnrs_profiler to distinguish it from the tnrs_profiler used by tnrs.tnrs_request(), which just profiles the current request
Aaron Marcuse-Kubitza
02:36 PM Revision 9521: bugfix: bin/tnrs_db: cumulative profiler: use len(names) instead of this_ct (cur.rowcount) in case the actual # rows fetched differed from the rowcount
Aaron Marcuse-Kubitza
02:32 PM Revision 9520: lib/tnrs.py: repeated_tnrs_request(): renamed to tnrs_request() since this is the function that should usually be used, to ensure that debugging information is output in the case of an error. (the TNRS request must be made again to output this information.)
Aaron Marcuse-Kubitza
02:30 PM Revision 9519: lib/tnrs.py: tnrs_request(): renamed to single_tnrs_request() to distinguish it from repeated_tnrs_request()
Aaron Marcuse-Kubitza
02:25 PM Revision 9518: bin/tnrs_db: removed no longer used $wait flag (which caused tnrs_db to wait max_pause for new rows to be added), because tnrs_db is now invoked automatically after each import by the import_scrub target (in inputs/input.Makefile) and does not need to run as a daemon. note that when scrub is invoked, it is possible that a previous datasource's import has already scrubbed the names for this import, because tnrs_db runs until all rows in tnrs_input_name are scrubbed.
this also removes clutter in tnrs_db, making it clearer what operations it performs that the library function tnrs.re... Aaron Marcuse-Kubitza
02:14 PM Revision 9517: bin/tnrs_db: removed no longer needed explicit population of the Time_submitted, which is now done automatically by the tnrs table. however, this requires starting the transaction *before* submitting data, so Time_submitted is correctly set to the submission time rather than the insertion time. the setting of the correct time can be tested by inserting `time.sleep(n_sec)` after the TNRS request and checking that the Time_submitted is close to the time tnrs_db was run instead of n_sec seconds later.
Aaron Marcuse-Kubitza
02:09 PM Revision 9516: bin/tnrs_db: start transaction *before* submitting data, so Time_submitted is correctly set to the submission time rather than the insertion time. these may differ by several minutes if TNRS is slow. the setting of the correct time can be tested by inserting `time.sleep(n_sec)` after the TNRS request, removing the explicit setting of Time_submitted, and checking that the Time_submitted is close to the time tnrs_db was run instead of n_sec seconds later.
Aaron Marcuse-Kubitza
02:05 PM Revision 9515: bugfix: bin/tnrs_db: wrap just the TNRS request and the storing of the response data in a function (undoing part of r9514), because the transaction start time for Time_submitted should not be until the TNRS request is actually made (it often takes several minutes to materialize the next set of input names on a full DB)
Aaron Marcuse-Kubitza
01:56 PM Revision 9514: bin/tnrs_db: Iterate over unscrubbed verbatim taxonlabels: put loop body in a function (which returns whether or not the loop should continue), so that the loop body can easily be wrapped in a transaction using sql.with_savepoint()
Aaron Marcuse-Kubitza
01:19 PM Revision 9513: inputs/.TNRS/schema.sql: tnrs.Time_submitted: set default to now() (the timestamp of the start of the current transaction, http://www.postgresql.org/docs/9.1/static/functions-datetime.html) so that it would automatically be populated when rows are added. note that because the start of the current transaction instead of the exact time at insertion is used, all rows inserted in the same transaction (e.g. as part of the same batch) will have the same value for this, linking them together.
Aaron Marcuse-Kubitza
01:10 PM Revision 9512: inputs/.TNRS/schema.sql: tnrs_populate_derived_fields(): renamed to tnrs_populate_fields() so it can be used to populate other fields as well
Aaron Marcuse-Kubitza
01:07 PM Revision 9511: bin/tnrs_db: removed no longer needed explicit appending of derived cols, and instead use append_csv()'s new support for importing CSVs whose columns are a subset of the full table
Aaron Marcuse-Kubitza
12:56 PM Revision 9510: bin/tnrs_db: ColInsertFilters: use the simpler literal value option for the mk_value param
Aaron Marcuse-Kubitza
12:55 PM Revision 9509: lib/csvs.py: ColInsertFilter: support using a literal value instead of a function for the mk_value param, since this is the most common use case
Aaron Marcuse-Kubitza
12:43 PM Revision 9508: lib/sql_io.py: append_csv(): support importing CSVs whose columns are a subset of the full table and/or in a different order. when the header exactly matches the columns, the explicit column list will still be omitted as an optimization. this uses code from r4927.
Aaron Marcuse-Kubitza
12:15 PM Revision 9507: bugfix: lib/runscripts/util.run: need to include sh/make.sh for all runscripts that use make-style commands
Aaron Marcuse-Kubitza
12:12 PM Revision 9506: *{.sh,run}: use new top_make instead of `make --directory="$top_dir"`
Aaron Marcuse-Kubitza
12:11 PM Revision 9505: lib/sh/make.sh: added top_make()
Aaron Marcuse-Kubitza
11:54 AM Revision 9504: inputs/import.stats.xls: Postprocessing: populated entries for analytical DB for last 4 imports, and for backup, backup test for last import. note that the combined import time for the last import is 3.5 days, compared to 3 days for the column-based import portion.
Aaron Marcuse-Kubitza

05/22/2013

11:47 PM Revision 9503: inputs/import.stats.xls: Postprocessing: added (empty) entries for analytical DB, backup, backup test
Aaron Marcuse-Kubitza

05/21/2013

11:18 PM Revision 9502: inputs/GBIF/Specimen/postprocess.sql, inputs/REMIB/Specimen/postprocess.sql: updated for providers in r9459, which adds TEX
Aaron Marcuse-Kubitza
11:10 PM Revision 9501: inputs/*/*/postprocess.sql: Remove institutions that we have direct data for: query to obtain list: updated for current schema
Aaron Marcuse-Kubitza
10:49 PM Revision 9500: inputs/import.stats.xls: Updated import times. GBIF has been refreshed (with the range modeling column subset), and column-based import now takes 3 days for 88.4 million rows.
Aaron Marcuse-Kubitza
10:27 PM Revision 9499: README.TXT: Full database import: added warning to perform *every single* step listed, to avoid breaking column-based import
Aaron Marcuse-Kubitza
10:26 PM Revision 9498: README.TXT: Full database import: Publish the new import: added warning to be sure you have done *every single* verification step before proceeding. otherwise, a previous valid import could incorrectly be overwritten with a broken one.
Aaron Marcuse-Kubitza
09:07 PM Revision 9497: bugfix: README.TXT: Full database import: To run TNRS/remake analytical DB: need to run `export version=<version>` *before* the command which uses it rather than after
Aaron Marcuse-Kubitza
08:26 PM Revision 9496: added backups/*.md5
Aaron Marcuse-Kubitza
08:22 PM Revision 9495: added backups/TNRS.2013-5-21.backup.md5
Aaron Marcuse-Kubitza
07:42 PM Revision 9494: README.TXT: Datasource setup: For MySQL inputs: For .sql exports: added steps to grant privileges to the bien user. the privileges list *excludes* UPDATE, DELETE, ALTER, DROP to prevent bugs in the import scripts from accidentally deleting data.
Aaron Marcuse-Kubitza
07:37 PM Revision 9493: inputs/.TNRS/schema.sql, data.sql: updated for new TNRS CSV columns (see bug at https://pods.iplantcollaborative.org/jira/browse/TNRS-183). note that these columns may eventually change back (comment by Naim at https://pods.iplantcollaborative.org/jira/browse/TNRS-183#comment-34444).
Aaron Marcuse-Kubitza
07:33 PM Revision 9492: README.TXT: Full database import: added steps to check that TNRS ran successfully, and fix errors (due to column changes in the TNRS CSV) if it didn't
Aaron Marcuse-Kubitza
07:24 PM Revision 9491: inputs/test_taxonomic_names/test_scrub: use sh's -e (errexit) mode so errors in an invoked script cause the script to abort instead of burying the error in more output
Aaron Marcuse-Kubitza
07:19 PM Revision 9490: inputs/test_taxonomic_names/test_scrub: documented that `make schemas/"$public"/uninstall` removes the previous results (since it may be confusing why it's prompting the user to uninstall the schema that is an output of the program)
Aaron Marcuse-Kubitza
07:16 PM Revision 9489: inputs/test_taxonomic_names/test_scrub: don't need to run the import twice anymore because the accepted names are now included in the tnrs_input_name view that TNRS runs on
Aaron Marcuse-Kubitza
07:09 PM Revision 9488: inputs/test_taxonomic_names/test_scrub: updated for current TNRS schema
Aaron Marcuse-Kubitza
06:47 PM Revision 9487: bugfix: inputs/test_taxonomic_names/test_scrub: unset $n so it doesn't limit the # rows. it is set to 2 in the default test environment, so must be unset for n-sensitive programs that should be unlimited.
Aaron Marcuse-Kubitza
06:40 PM Revision 9486: inputs/test_taxonomic_names/test_scrub: updated for current TNRS schema
Aaron Marcuse-Kubitza
01:44 PM Revision 9485: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.table/make(): also include the exported plant_fraction herbaria
Aaron Marcuse-Kubitza
01:43 PM Revision 9484: inputs/GBIF/raw_occurrence_record/run: added herbaria_filter.plant_fraction.csv_/make(), which exports the plant_fraction herbaria whose plant_fraction >= 0.8
Aaron Marcuse-Kubitza
01:42 PM Revision 9483: inputs/GBIF/raw_occurrence_record/run: added plant_fraction.table/make(), which contains the plant fraction for each herbarium
Aaron Marcuse-Kubitza
01:37 PM Revision 9482: lib/sh/db.sh: added mk_drop()
Aaron Marcuse-Kubitza
01:00 PM Revision 9481: lib/sh/util.sh: to_file(): log $stdout so users can tell which file is being created by the command. for some reason, can't use `local redirs=(">$stdout")` because the redirections don't seem to be applied. can't yet use `log+ -2 echo_vars stdout` because log+ does not yet support negative adjustments (they cause PS4 to be emptied out before being re-prepended to).
Aaron Marcuse-Kubitza
12:54 PM Revision 9480: bugfix: lib/sh/util.sh: log+(): adjustment < 0: need to enclose -$1 in $(()) so it gets evaluated before being used as an array index
Aaron Marcuse-Kubitza
12:16 PM Revision 9479: lib/sh/local.sh: psql(): documented that --output is actually for query *results*, not echoed statements (and thus must be redirected back to fd 1 while fd 1 with the statements gets sent to the logging port)
Aaron Marcuse-Kubitza
12:14 PM Revision 9478: lib/sh/local.sh: psql(): documented why can't use fd 11
Aaron Marcuse-Kubitza
12:09 PM Revision 9477: lib/sh/local.sh: use @redirs instead of manual redirection to set up --output fd, so that the redirection will be echoed along with the command. for some reason, this requires switching to fd 13 instead of 11, because fd 11 gives a "/dev/fd/11: Bad file descriptor" error when 11 is set with exec right before the command instead of on the subshell the command is executed in. (13 was chosen rather than 12 because *2 is for errors, while *3 (or 3*) is for logging.)
Aaron Marcuse-Kubitza
04:18 AM Revision 9476: bugfix: lib/sh/db.sh: pg_export_table_to_dir_no_header(): inlined $(pg_header) so setting $cols wouldn't affect pg_export_table_no_header(), which uses it as a kw param
Aaron Marcuse-Kubitza

05/20/2013

10:44 PM Revision 9475: bugfix: lib/sh/util.sh: to_file(): require_not_exists check: missing `test` in `if "$if_not_exists"`
Aaron Marcuse-Kubitza
10:39 PM Revision 9474: lib/sh/util.sh: command(): log the function call using echo_func to assist debugging. (use a higher log_level because it's internal.)
Aaron Marcuse-Kubitza
09:29 PM Revision 9473: lib/sh/util.sh: command(): support custom redirections, which will be echoed along with the command
Aaron Marcuse-Kubitza
08:48 PM Revision 9472: lib/sh/util.sh: to_file(): reworded confusing || conditional for require_not_exists into an if statement
Aaron Marcuse-Kubitza
08:21 PM Revision 9471: bugfix: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.table/make(): need to use append=1 with mysql_import so the output table doesn't get re-truncated when additional parts are added
Aaron Marcuse-Kubitza
07:28 PM Revision 9470: bugfix: lib/sh/db.sh: load new aliases before mk_select(), which uses mk_table_esc
Aaron Marcuse-Kubitza
07:27 PM Revision 9469: lib/runscripts/table.run: include make.sh so runscripts based on it can use make-related utils
Aaron Marcuse-Kubitza
06:52 PM Revision 9468: lib/sh/db.sh: added mk_select() and use it in mk_select_var
Aaron Marcuse-Kubitza
06:46 PM Revision 9467: lib/sh/db.sh: added limit() and use it instead of `${limit:+LIMIT $limit}`
Aaron Marcuse-Kubitza
06:44 PM Revision 9466: lib/sh/db.sh: added mysql_truncate() and use it instead of `mk_truncate|mysql_ANSI`
Aaron Marcuse-Kubitza
06:42 PM Revision 9465: lib/sh/db.sh: truncate(): renamed to mk_truncate() because it actually just creates a TRUNCATE statement, rather than also executing it
Aaron Marcuse-Kubitza
06:38 PM Revision 9464: lib/sh/db.sh: use_local/use_remote: unset $prefix after using it so it isn't unintentionally applied as a kw param for a later function
Aaron Marcuse-Kubitza
04:18 PM Revision 9463: lib/sh/db.sh: mk_select: renamed to mk_select_var since it actually sets a var in the local context rather than returning a query
Aaron Marcuse-Kubitza
03:40 PM Revision 9462: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.table/make(): specify the different parts used to create the table in an array
Aaron Marcuse-Kubitza
03:19 PM Revision 9461: inputs/GBIF/raw_occurrence_record/run: renamed herbaria_filter.csv_ to herbaria_filter.ih.csv_ to allow for other tables that get combined into herbaria_filter
Aaron Marcuse-Kubitza
03:13 PM Revision 9460: bugfix: lib/sh/db.sh: mk_select: ensure newline before LIMIT clause, in case caller provided custom query which did not have trailing newline
Aaron Marcuse-Kubitza

05/17/2013

06:00 PM Revision 9459: bugfix: mappings/VegCore-VegBIEN.csv: place.geovalid: added missing /1 after _alt
Aaron Marcuse-Kubitza
05:55 PM Revision 9458: bugfix: lib/sql.py: parse_exception(): typed_name_re: added back matching of names without "", since these are used by some error messages (ones that contain () after the function name)
Aaron Marcuse-Kubitza
05:41 PM Revision 9457: bugfix: lib/sql.py: parse_exception(): typed_name_re: need to allow " within the matched name, since there are now "" around the entire identifer that was passed to Postgres, which may itself include " . always require "" around the matched name, to ensure that the whole name is matched by .+? e.g. when followed by () for a function call. the version of Postgres we currently use apparently no longer has error messages without the "", so we don't need a separate regexp for quoted and unquoted names.
Aaron Marcuse-Kubitza
03:43 PM Revision 9456: lib/sh/db.sh: mysql_import(): automatically ensure the table is empty (i.e. using truncate()), unless append=1 is specified. extra calls to truncate() now that this happens automatically have also been removed.
Aaron Marcuse-Kubitza
01:13 PM Revision 9455: bin/map: by_col: ensure verbosity is at least 2 in live mode (using new ints.set_min() instead of max() for clarity). documented that live column-based import MUST be run with verbosity 2+ (3 preferred) to provide debugging information for often-complex errors. without this, debugging is effectively impossible.
Aaron Marcuse-Kubitza
01:08 PM Revision 9454: added lib/ints.py with renamings of max()->set_*min*(), min()->set_*max*() for easier understandability of the set-ceiling/set-floor use cases of min()/max()
Aaron Marcuse-Kubitza
12:57 PM Revision 9453: bin/map: Set default verbosity: by_col: documented that showing all queries is primarily to assist *debugging*, not profiling
Aaron Marcuse-Kubitza
11:59 AM Revision 9452: lib/sh/util.sh: logging: named it `log++`
Aaron Marcuse-Kubitza
11:59 AM Revision 9451: lib/sh/util.sh: logging: verbosities: level 0: documented that log++ also suppresses external command output for full support of cron jobs
Aaron Marcuse-Kubitza
11:57 AM Revision 9450: lib/sh/util.sh: logging: documented `make` equivalents of the various verbosities, where available. (many of the verbosities, such as level 1, are sorely needed in make to avoid excessive output.)
# verbosities (and `make` equivalents):
# 0: just print errors. useful for cron jobs.
# vs. make: equivalent to --...
Aaron Marcuse-Kubitza
04:03 AM Revision 9449: lib/sh/util.sh: die_e(): benign errors: increase log_level so that a benign non-zero exit status will only be displayed at debug verbosities (2+) (it is confusing otherwise)
Aaron Marcuse-Kubitza
03:36 AM Revision 9448: lib/sh/util.sh: try(): always run the command with benign_error=1 so that any die_e() doesn't prematurely indicate that a particular exit status was an error
Aaron Marcuse-Kubitza
03:34 AM Revision 9447: lib/sh/util.sh: die_e(): support benign errors using $benign_error flag that should be logged as info messages instead of errors
Aaron Marcuse-Kubitza
03:30 AM Revision 9446: lib/sh/util.sh: die(): documented that msg can't use $() (because it would reset $?)
Aaron Marcuse-Kubitza
03:19 AM Revision 9445: inputs/bien_web/observation/VegBIEN.csv, unmapped_terms.csv: regenerated
Aaron Marcuse-Kubitza
03:01 AM Revision 9444: lib/sh/util.sh: command(): 2>&$err_fd: add to _redirs *after* echoing command so it isn't echoed at the end of every command (since this redirection is frequently applied)
Aaron Marcuse-Kubitza
02:55 AM Revision 9443: lib/sh/util.sh: sed: use case statement instead of test to determine flag letter, to easily allow matching multiple `uname` OSes or adding additional flag letters
Aaron Marcuse-Kubitza
02:46 AM Revision 9442: lib/sh/util.sh: die(): documented that its msg can use $?, because it has not yet been overridden by another command
Aaron Marcuse-Kubitza
02:45 AM Revision 9441: lib/sh/util.sh: die_e(): use die(), which performs the necessary save_e/rethrow. this requires using $? instead of $e for the exit status, because $e has not yet been set.
Aaron Marcuse-Kubitza
02:42 AM Revision 9440: lib/sh/util.sh: inlined log_e() into die_e() because that's the only place it's used
Aaron Marcuse-Kubitza
02:37 AM Revision 9439: lib/sh/util.sh: command(): print "command exited with error" message using new die_e() if command returns false. this requires removing manual die_e()/log_e() calls elsewhere.
Aaron Marcuse-Kubitza
02:34 AM Revision 9438: lib/sh/util.sh: command(): moved increase of indent inside () so that error-handling statements after () will use the outer log_level
Aaron Marcuse-Kubitza
02:31 AM Revision 9437: lib/sh/util.sh: added die_e(), which logs that a command exited with an error
Aaron Marcuse-Kubitza
02:18 AM Revision 9436: lib/sh/util.sh: command(): determine redirections before echoing the command so they can be logged along with the command, instead of as separate exec statements. (these had a higher log_level to avoid cluttering the output with `exec` lines, which usually suppressed the redirections completely.) inline the command__set_fds() nested func so the redirections are all in one place.
Aaron Marcuse-Kubitza
01:54 AM Revision 9435: lib/sh/util.sh: use simpler `if can_log; then indent; fi` instead of `can_log && indent || true`. however, the `&& indent || true` syntax is still required in aliases such as echo_func which need to allow prefixing the command with a wrapper command or kw param assignments.
Aaron Marcuse-Kubitza

05/16/2013

09:28 PM Revision 9434: inputs/GBIF/raw_occurrence_record/run: dynamically generate herbaria_filter.csv_ from herbaria.ih in new target herbaria_filter.csv_/make()
Aaron Marcuse-Kubitza
09:27 PM Revision 9433: inputs/GBIF/raw_occurrence_record/run: store the herbaria filter in a MySQL table loaded from a CSV instead of getting it from a hardcoded list of IN (...) values
Aaron Marcuse-Kubitza
09:24 PM Revision 9432: lib/sh/db.sh: added truncate()
Aaron Marcuse-Kubitza
09:23 PM Revision 9431: lib/sh/make.sh: set_make_vars: set $target_stem
Aaron Marcuse-Kubitza
08:49 PM Revision 9430: lib/sh/db.sh: added mysql_import()
Aaron Marcuse-Kubitza
07:02 PM Revision 9429: lib/sh/db.sh: removed no longer used mk_esc_name()
Aaron Marcuse-Kubitza
07:01 PM Revision 9428: lib/runscripts/table.run: don't mk_esc_name schema, table because these will be mk_esc_name'd by functions that use them
Aaron Marcuse-Kubitza
06:55 PM Revision 9427: lib/sh/local.sh: psql(): use $schema_esc, $table_esc instead of just putting $schema, $table in ""
Aaron Marcuse-Kubitza
06:48 PM Revision 9426: lib/sh/db.sh: mk_esc_name_alias(): don't overwrite an already-defined $*_esc, to allow the user to provide an already-escaped value (such as a schema-qualified table) directly
Aaron Marcuse-Kubitza
06:38 PM Revision 9425: lib/sh/util.sh: rtrim(): increase the log_level of sed to 4+ instead of 2+ because it is usually run as part of a var assignment, and should therefore have a lower log_level than echo_vars
Aaron Marcuse-Kubitza
06:32 PM Revision 9424: lib/sh/db.sh: mk_esc_name_alias(): echo_vars the *_esc var when it's set
Aaron Marcuse-Kubitza
06:31 PM Revision 9423: lib/sh/db.sh: added mk_esc_name_alias() and use it to create mk_schema_esc, mk_table_esc
Aaron Marcuse-Kubitza
05:55 PM Revision 9422: lib/sh/db.sh: mysql(): run with --local-infile=1
Aaron Marcuse-Kubitza
05:48 PM Revision 9421: bugfix: lib/sh/db.sh: log_sql(): use can_log() instead because the verbosity now gets decremented as the log_level increases, so the threshold to compare to is 0 instead of 2
Aaron Marcuse-Kubitza
05:46 PM Revision 9420: lib/sh/util.sh: added set_default()
Aaron Marcuse-Kubitza
05:45 PM Revision 9419: lib/sh/util.sh: rtrim(): run at higher log_level so that sed command is not normally echoed
Aaron Marcuse-Kubitza
04:40 PM Revision 9418: inputs/GBIF/raw_occurrence_record/run: renamed herbaria.sql to herbaria.data.sql so it wouldn't be added to svn by `make inputs/GBIF/raw_occurrence_record/add` or `make inputs/add`
Aaron Marcuse-Kubitza
04:38 PM Revision 9417: inputs/input.Makefile: $(svnFiles): also exclude *.data.sql, which should never be in svn
Aaron Marcuse-Kubitza
04:27 PM Revision 9416: schemas/vegbien.sql: cultivated_family_locations: documented that table is from sftp://nimoy.nceas.ucsb.edu/home/bien/bien2_scripts/geoscrub/cultivated/cult_by_taxon/flag_by_taxa.inc (i.e. not generated by a function)
Aaron Marcuse-Kubitza
04:15 PM Revision 9415: schemas/vegbien.sql: place.geovalid: added latLongDomainValid to the values to _and together
Aaron Marcuse-Kubitza
04:09 PM Revision 9414: schemas/vegbien.sql: place.geovalid: require it to be NOT NULL so that it's always a 2-valued boolean (but default it to false since it's not a required field)
Aaron Marcuse-Kubitza
04:06 PM Revision 9413: mappings/VegCore-VegBIEN.csv: place.geovalid: use false instead of NULL
Aaron Marcuse-Kubitza
03:46 PM Revision 9412: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): exclude deleted rows (i.e. where the deleted timestamp is non-NULL)
Aaron Marcuse-Kubitza
03:42 PM Revision 9411: inputs/GBIF/raw_occurrence_record/header.csv: regenerated using ./run. since the table is reimported as a CSV, it uses bin/csv2db, which prepends an additional row_num column.
Aaron Marcuse-Kubitza
03:09 PM Revision 9410: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): remove explicit cols list to include all cols. the file size of the generated table.tsv will increase by ~3x, but should remain reasonably-sized compared to our available disk space.
Aaron Marcuse-Kubitza
03:04 PM Revision 9409: bugfix: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): need \ line continuation after vars so they only apply to the command rather than being set as global vars
Aaron Marcuse-Kubitza
03:02 PM Revision 9408: bugfix: lib/runscripts/table.run: load_data(): use new $verbosity_min instead of running `verbosity_min` so that the command name logging is not output with the new verbosity
Aaron Marcuse-Kubitza
02:59 PM Revision 9407: lib/sh/util.sh: added $verbosity_min to set a `verbosity_min` value *after* the command name, etc. has been logged, so that the logging itself is not output with the new verbosity
Aaron Marcuse-Kubitza
02:38 PM Revision 9406: schemas/vegbien.sql: range_modeling_input: include only plants (i.e. rows with higher_plant_group IS NOT NULL)
Aaron Marcuse-Kubitza
02:36 PM Revision 9405: schemas/vegbien.sql: range_modeling_input: added higher_plant_group, for use in restricting rows to plants
Aaron Marcuse-Kubitza
02:24 PM Revision 9404: inputs/.geoscrub/geoscrub_output/map.csv: *validity: added definitions of the numeric codes from _src/README.TXT
Aaron Marcuse-Kubitza
01:11 PM Revision 9403: added planning/workflow/validation/GeoDistKM.sql.txt
Aaron Marcuse-Kubitza
12:42 PM Revision 9402: planning/goals/BIEN3_derived_data_products.docx: updated to most recent version from Brad's e-mail on 2013-4-16
Aaron Marcuse-Kubitza
11:18 AM Revision 9401: /README.TXT: Full database import: before running screen: added `unset TMOUT` because TMOUT (autologout) causes screen to exit even with background processes active
Aaron Marcuse-Kubitza
11:17 AM Revision 9400: /README.TXT: Maintenance: added things to put in your .profile on a live machine (e.g. vegbiendev). in particular, you *MUST NOT* have a TMOUT (autologout) set, because this causes screen to exit even if background processes (e.g. from column-based import) are running
Aaron Marcuse-Kubitza
11:15 AM Revision 9399: /README.TXT: Maintenance: added things to put in your .profile on a live machine (e.g. vegbiendev). in particular, you *MUST NOT* have a TMOUT (autologout) set, because this causes screen to exit even if background processes (e.g. from column-based import) are running
Aaron Marcuse-Kubitza
06:15 AM Revision 9398: bugfix: lib/runscripts/table.run: load_data(): need to ensure the verbosity is at least 3 because the install logs require verbose output. (3 is the default for the installer, but is overridden by the runscripts, which instead set the default to 1.)
Aaron Marcuse-Kubitza
06:13 AM Revision 9397: lib/sh/util.sh: logging: added verbosity_min()
Aaron Marcuse-Kubitza
06:00 AM Revision 9396: lib/sh/util.sh: logging: added verbosity_int() and use it instead of `round_down "$verbosity"`
Aaron Marcuse-Kubitza
02:08 AM Revision 9395: bugfix: lib/sh/util.sh: log+ alias: don't expand next word because it's not a cmd
Aaron Marcuse-Kubitza
01:59 AM Revision 9394: lib/runscripts/table.run: $schema, $table: log the `cd` used to calculate the value at log_level 3 instead of 1 (note that the cd() function call for this will be logged at log_level 5)
Aaron Marcuse-Kubitza
01:51 AM Revision 9393: lib/sh/util.sh: cd(): trace-log the cd() function call itself (at log_level 3) in addition to the cd builtin call
Aaron Marcuse-Kubitza

05/15/2013

11:28 PM Revision 9392: lib/runscripts/table.run: import(): added step to load the data into the staging table before postprocessing it
Aaron Marcuse-Kubitza
11:27 PM Revision 9391: inputs/GBIF/raw_occurrence_record/run: moved table.tsv.md5/make() and invocation of it to inputs/GBIF/table.run because it's general to all tables (which would all use table.tsv for this datasource). use $target_filename in calling table.tsv.md5/make from table.tsv/make.
Aaron Marcuse-Kubitza
11:21 PM Revision 9390: bugfix: lib/sql.py: parse_exception(): typed_name_re: need to ensure that full name is matched rather than just first character
Aaron Marcuse-Kubitza
10:26 PM Revision 9389: web/links/index.htm: updated to Firefox bookmarks. Mountain Lion upgrade: added Python psycopg2, Python OrderedDict, X11.
Aaron Marcuse-Kubitza
07:51 PM Revision 9388: lib/runscripts/table.run: table_make(): run canon_rel_path on the datasrc dir so it's displayed as the direct path to it, without ..
Aaron Marcuse-Kubitza
07:47 PM Revision 9387: lib/runscripts/table.run: table_make(): moved comment about "${@/#/$table/}" to right after the line it describes
Aaron Marcuse-Kubitza
07:44 PM Revision 9386: lib/runscripts/table.run: input_make(): renamed to table_make() to make it clear that the target names are relative to the table subdir itself, not the datasrc dir. it was previously called input_make because it used inputs/input.Makefile directly, but now will use any Makefile in the datasrc dir.
Aaron Marcuse-Kubitza
06:44 PM Revision 9385: added inputs/GBIF/Makefile, which links to ../input.Makefile, to allow running make directly in the datasrc dir (i.e. without --makefile=.../input.Makefile). this is *required* by the runscripts.
Aaron Marcuse-Kubitza
03:34 PM Revision 9384: inputs/GBIF/raw_occurrence_record/run: table.tsv.md5/make(): don't add extra .md5 extension to $target_filename because it already has the extension as part of the target name (now that this command is run in its own make target rather than in table.tsv/make())
Aaron Marcuse-Kubitza
03:30 PM Revision 9383: lib/sql_gen.py: import OrderedDict from collections instead of ordereddict for Mac 10.8 Mountain Lion upgrade
Aaron Marcuse-Kubitza
02:29 PM Revision 9382: lib/sh/util.sh: logging: renamed $log_level_indent to $log_indent_step to avoid confusion with the log_level, which is a different kind of indent (using + signs instead of |s)
Aaron Marcuse-Kubitza
02:26 PM Revision 9381: lib/sh/util.sh: logging: also export PS4, because it follows verbosity and therefore also needs to be propagated to invoked commands
Aaron Marcuse-Kubitza
02:21 PM Revision 9380: lib/sh/util.sh: can_log(): support decimal verbosities using round_down()
Aaron Marcuse-Kubitza
02:21 PM Revision 9379: lib/sh/util.sh: always use echo_export instead of export, even when the verbosity at load time would suppress output, because the verbosity may actually increase during the script due to log-- calls, etc., and vars should then still be echoed as expected
Aaron Marcuse-Kubitza
02:17 PM Revision 9378: lib/sh/util.sh: log+: inlined PS4_prefix_n alias because there is now room for it
Aaron Marcuse-Kubitza
02:16 PM Revision 9377: lib/sh/util.sh: log+: $verbosity: support decimal verbosities (but not decimal log_levels) by using new float+int
Aaron Marcuse-Kubitza
02:13 PM Revision 9376: lib/sh/util.sh: removed no longer used log-. use log+ with a negative argument instead.
Aaron Marcuse-Kubitza
02:12 PM Revision 9375: bugfix: lib/sh/util.sh: log+: PS4 when $1 < 0: need to negate $1 because now it's a negative number
Aaron Marcuse-Kubitza
02:06 PM Revision 9374: lib/sh/util.sh: log--: use log+ with -1 instead of log- so we don't need a separate log- function
Aaron Marcuse-Kubitza
02:04 PM Revision 9373: lib/sh/util.sh: logging: log+: support negative log_level adjustments. log-: use log+ with the negative of its argument
Aaron Marcuse-Kubitza
01:48 PM Revision 9372: web/links/index.htm: updated to Firefox bookmarks. sorted MySQL and PostgreSQL sections.
Aaron Marcuse-Kubitza
01:42 PM Revision 9371: web/links/index.htm: updated to Firefox bookmarks
Aaron Marcuse-Kubitza
01:39 PM Revision 9370: web/links/index.htm: Mac 10.8 Mountain Lion > PostgreSQL: appended tab to name to disambiguate it from the general PostgreSQL section
Aaron Marcuse-Kubitza
01:33 PM Revision 9369: bugfix: web/links/index.htm.run: prepend __ to the HTML anchors of bookmarks toolbar links so they don't shadow/override folders of the same name: also renamed the corresponding self-anchor hyperlinks
Aaron Marcuse-Kubitza
01:32 PM Revision 9368: bugfix: web/links/index.htm.run: prepend __ to the HTML anchors of bookmarks toolbar links so they don't shadow/override folders of the same name: need to replace *all* occurrences (using /g option to sed) to include both HTML anchors on the line
Aaron Marcuse-Kubitza
01:26 PM Revision 9367: web/links/index.htm.run: prepend __ to the HTML anchors of bookmarks toolbar links so they don't shadow/override folders of the same name
Aaron Marcuse-Kubitza
01:23 PM Revision 9366: lib/sh/util.sh: sed: changed it to an alias so it will also be expanded when passed to an external command (like in_place) that can only run an executable, not a shell function (this occurs as long as the external command is defined as an alias which ends in space, to alias-expand the next word). added associated $sed_cmd var for cases when there is no alias wrapper around the external command, and the literal alias body must be used instead.
Aaron Marcuse-Kubitza
11:02 AM Revision 9365: lib/sh/util.sh: log+/-(): setting verbosity: added space around operators to support negative numbers
Aaron Marcuse-Kubitza
10:55 AM Revision 9364: lib/sh/util.sh: added float functions (esp. float+int()) for dealing with decimal verbosities used by sql.py and column-based import
Aaron Marcuse-Kubitza
10:53 AM Revision 9363: inputs/GBIF/raw_occurrence_record/run: inputs/GBIF/raw_occurrence_record/run: added check_target_exists so you know why make skipped the file (for other, non-silent targets, it would also avoid make's verbose output when the file exists)
Aaron Marcuse-Kubitza
10:38 AM Revision 9362: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): moved making of table.tsv.md5 to separate function
Aaron Marcuse-Kubitza
10:30 AM Revision 9361: lib/sh/util.sh: $log_fd: indicated that this is only *initially* stderr (however, the new port will just use stderr if it's not redirected separately)
Aaron Marcuse-Kubitza
10:28 AM Revision 9360: lib/sh/util.sh: verbosities: 3: added that this includes values of kw params
Aaron Marcuse-Kubitza
10:25 AM Revision 9359: lib/sh/util.sh: logging: added description of the verbosities available, including what each one does and what it's useful for:
# verbosities:
# 0: just print errors. useful for cron jobs.
# 1: also external commands run. useful for running at t...
Aaron Marcuse-Kubitza
10:11 AM Revision 9358: lib/sh/util.sh: command__set_fds(): echo the >& line at a higher log_level, because this information (i.e. which fd is used by the command for logging) is primarily for debugging and should not normally be printed
Aaron Marcuse-Kubitza
10:00 AM Revision 9357: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): also add md5 sum for table.tsv
Aaron Marcuse-Kubitza
09:50 AM Revision 9356: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added back filter kw args, which had gotten deleted in a commit without update (although actually, svn should not allow a commit without update, so the working copy may have gotten corrupted)
Aaron Marcuse-Kubitza
09:27 AM Revision 9355: lib/runscripts/table.run: input_make(): added $silent flag which turns on make's --silent option
Aaron Marcuse-Kubitza
09:25 AM Revision 9354: lib/sh/make.sh: set_make_vars: usage: added "use $target" to indicate that vars are made available by this alias
Aaron Marcuse-Kubitza
09:24 AM Revision 9353: lib/sh/make.sh: inline_make(): take the script stdin from caller-provided stdin and get make's stdin from global stdin, so that the caller can just use <<'EOF' rather than having to include a specific fd before the <<
Aaron Marcuse-Kubitza
08:46 AM Revision 9352: lib/runscripts/table.run: input_make(): documented that it requires a Makefile in the datasrc dir containing `include ../input.Makefile
Aaron Marcuse-Kubitza
08:41 AM Revision 9351: lib/runscripts/table.run: input_make(): added comment explaining that "${@/#/$table/}" replaces the empty str at the beginning of str (/#) with $table/
Aaron Marcuse-Kubitza
08:38 AM Revision 9350: lib/runscripts/table.run: put functions on one line where possible
Aaron Marcuse-Kubitza
08:35 AM Revision 9349: lib/runscripts/table.run: input_make(): use the local make() function instead of external make directly, because it sets $cmd_log_fd appropriately to ensure that all the echoed make commands get properly logged to stdlog/stderr (stdlog is fd 30 when it's redirected to a file)
Aaron Marcuse-Kubitza
08:22 AM Revision 9348: inputs/GBIF/raw_occurrence_record/run table.tsv/make() and functions used by it: added usage comments for cmd line usage, caller usage, and declaring function usage
Aaron Marcuse-Kubitza
07:29 AM Revision 9347: web/links/index.htm: updated to Firefox bookmarks
Aaron Marcuse-Kubitza
07:27 AM Revision 9346: web/links/index.htm: updated to Firefox bookmarks
Aaron Marcuse-Kubitza
07:00 AM Revision 9345: web/links/index.htm: removed no longer needed - at the beginning of every folder's description
Aaron Marcuse-Kubitza
07:00 AM Revision 9344: lib/Firefox_bookmarks.reformat.csv: label page's description: don't do this for folders (i.e. descriptions preceded by an <H3> tag) because their descriptions are always author-added rather than from a web page. this avoids needing to add a - at the beginning of every folder's description.
Aaron Marcuse-Kubitza

05/14/2013

09:43 PM Revision 9343: web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: put fixes subdirs in order.
Aaron Marcuse-Kubitza
09:42 PM Revision 9342: web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: put fixes subdirs in order.
Aaron Marcuse-Kubitza
08:56 PM Revision 9341: web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: added fixes for Apache ~/Sites dirs, Apache PHP
Aaron Marcuse-Kubitza
05:55 PM Revision 9340: web/links/index.htm: updated to Firefox bookmarks. added bookmarks for upgrading Mac OS X to 10.8 Mountain Lion. (WARNING: DO NOT upgrade unless you are prepared to fix several programs broken by the upgrade: svn, PostgreSQL. instructions are in the corresponding bookmark subdirs. these programs will be COMPLETELY UNAVAILABLE until they are manually fixed!)
Aaron Marcuse-Kubitza

05/13/2013

07:43 AM Revision 9339: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): cols: also include scientific_name, which is preferable as a TNRS input because it also contains lower ranks
Aaron Marcuse-Kubitza
07:40 AM Revision 9338: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): cols: also include id, institution_code, collection_code, catalogue_number
Aaron Marcuse-Kubitza
07:38 AM Revision 9337: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added filter for institution_codes in herbaria.ih (in PostgreSQL)
Aaron Marcuse-Kubitza
07:19 AM Revision 9336: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added column subset (from http://vegpath.org/twiki/bin/view/Main/ConfCall20130509#subsetting_strategy > include)
Aaron Marcuse-Kubitza
07:14 AM Revision 9335: lib/sh/db.sh: mk_select: constructed queries: in ${var:+if_true} syntax, put the newline at the end of the if_true value instead of the beginning, so that each ${var:+if_true} expression starts at the beginning of a line
Aaron Marcuse-Kubitza
07:10 AM Revision 9334: lib/sh/db.sh: mk_select: constructed queries: support custom columns list using $cols
Aaron Marcuse-Kubitza
07:07 AM Revision 9333: lib/sh/db.sh: mk_select: constructed queries: support WHERE clause using $filter
Aaron Marcuse-Kubitza
06:52 AM Revision 9332: lib/sh/util.sh: echo_stdin(): don' increase the log_level, because the input being sent to the command (which is usually a set of interpreted commands itself) is necessary to fully know what action is being performed
Aaron Marcuse-Kubitza
06:30 AM Revision 9331: bugfix: lib/sh/local.sh: add a manual errexit for $() exprs by embedding them in just a var assignment (*without* local or declare), whose exit status will then equal the of the $(). a `|| return` also needs to be added because errexit does not work on assignment statements. this commit adds them for func_loc(), echo_func(), canon_rel_path(), set_paths(), save_cache, cached realpath(), local.sh global vars
Aaron Marcuse-Kubitza
06:14 AM Revision 9330: lib/sh/util.sh: shell-variable-based caching: usage: updated alias names
Aaron Marcuse-Kubitza
05:33 AM Revision 9329: web/links/index.htm: updated to Firefox bookmarks. regenerating also removed extra <a name=...> tags that were added when running index.htm.run on an already-processed index.htm.
Aaron Marcuse-Kubitza
05:19 AM Revision 9328: bugfix: lib/sh/util.sh: `set -o`: need a -o before *every* option to set
Aaron Marcuse-Kubitza
05:10 AM Revision 9327: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' names: hyperlink the added anchors as clickable paragraph marks (like Redmine), which take you to the HTML anchor. this is analogous to the clickable folder names which take you to *their* anchors.
Aaron Marcuse-Kubitza
04:52 AM Revision 9326: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' *names*, in addition to their URLs
Aaron Marcuse-Kubitza
04:47 AM Revision 9325: bugfix: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' URLs: use .*? instead of .* to match the contents of the <A> tag before the HREF
Aaron Marcuse-Kubitza
04:45 AM Revision 9324: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links using the URL itself as the anchor. these can be used to link to the comments attached *to* a bookmark in the bookmarks page, rather than to the bookmark's destination.
Aaron Marcuse-Kubitza
04:43 AM Revision 9323: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links using the URL itself as the anchor. these can be used to link to the comments attached *to* a bookmark in the bookmarks page, rather than to the bookmark's destination.
Aaron Marcuse-Kubitza
04:35 AM Revision 9322: web/links/index.htm: updated to Firefox bookmarks
Aaron Marcuse-Kubitza
04:24 AM Revision 9321: lib/sh/util.sh: `set -o`: added pipefail option, to ensure that exit statuses (esp. for errexit) also work with pipelines (a|b)
Aaron Marcuse-Kubitza
04:13 AM Revision 9320: lib/sh/util.sh: use new log+ with a numeric arg instead of multiple calls to log++
Aaron Marcuse-Kubitza
04:06 AM Revision 9319: lib/sh/util.sh: use new log+ with a numeric arg instead of multiple calls to log++. use the command form of log-- (`log-- echo_func`) to counter the normal log++ performed by echo_func, for cases when the function name is descriptive and should be output at the same log_level as the commands it runs.
Aaron Marcuse-Kubitza

05/12/2013

08:33 PM Revision 9318: *{.sh,run}: removed extra space between function name and ()
Aaron Marcuse-Kubitza
08:13 PM Revision 9317: lib/sh/util.sh: echo_func(): take the FUNCNAME as an argument (auto-added in the echo_func alias) instead of getting it from the FUNCNAME array (which would have produced an inaccurate value if another function call (such as log++) intervened between the caller and echo_func())
Aaron Marcuse-Kubitza
08:04 PM Revision 9316: lib/sh/util.sh: echo_func(): display where the function was declared (using new func_loc()) instead of where echo_func() was called from. this is more intuitive when debugging, becaues the line # is where the function starts. it also helps remove the dependency on the FUNCNAME/BASH_* arrays, which would produce an inaccurate value if another function call (such as log++) intervened between the caller and echo_func().
Aaron Marcuse-Kubitza
07:57 PM Revision 9315: lib/sh/util.sh: added func_loc(), which gets where a function was declared in the format file:line, and helper alias set_func_loc
Aaron Marcuse-Kubitza
07:09 PM Revision 9314: lib/sh/util.sh: str2varname(): use bash's internal ${var//glob/repl} syntax, which supports character classes ([a-z], etc.), instead of sed (which is slower because it's an external command and uses regexps)
Aaron Marcuse-Kubitza
07:00 PM Revision 9313: lib/sh/util.sh: added support for caching realpath() using the new shell-variable-based caching. this can be enabled via $realpath_cache and defaults to off because it's currently slower than without. note that the cache needs to be cleared in cd() because relative paths will become invalid.
Aaron Marcuse-Kubitza
06:48 PM Revision 9312: lib/sh/util.sh: added str2varname() and use it in include_guard_var()
Aaron Marcuse-Kubitza
06:47 PM Revision 9311: lib/sh/util.sh: added shell-variable-based caching functions
Aaron Marcuse-Kubitza
05:08 PM Revision 9310: lib/sh/util.sh: canon_rel_path(): use $PWD instead of the slower $(pwd -P) now that $PWD has symlinks expanded
Aaron Marcuse-Kubitza
05:06 PM Revision 9309: lib/sh/util.sh: expand symlinks in $PWD: moved it before canon_rel_path(), which will require it
Aaron Marcuse-Kubitza
05:03 PM Revision 9308: lib/sh/util.sh: expand symlinks in $PWD using `cd -P .` so it matches the output of realpath
Aaron Marcuse-Kubitza
05:01 PM Revision 9307: lib/sh/util.sh: cd(): documented what -P option does and why it's used
Aaron Marcuse-Kubitza
04:55 PM Revision 9306: lib/sh/util.sh: cd(): always run cd with -P (expand symlinks) so that $PWD will contain a canonical path which matches the output of `readlink -f`. however, don't display the added -P in the logging output because it distracts from the directory being changed to.
Aaron Marcuse-Kubitza
04:29 PM Revision 9305: lib/sh/util.sh: logging: added log+/-, which take a variable log_level step, and use them in log++/--
Aaron Marcuse-Kubitza
04:16 PM Revision 9304: lib/sh/util.sh: repeat(): for simplicity and speed, just append to a local string var (and echo the result at the end) instead of using printf
Aaron Marcuse-Kubitza
04:12 PM Revision 9303: bugfix: lib/sh/util.sh: repeat(): need to use %s instead of %q (escaped string) in printf
Aaron Marcuse-Kubitza
03:59 PM Revision 9302: lib/sh/util.sh: logging: log++/--: comment about the different uses can be performed with one function: clarified that *without a cmd*, "$@" expands to nothing and assignments are applied to the caller
Aaron Marcuse-Kubitza
03:00 PM Revision 9301: lib/sh/util.sh: log++/--: support running a command with the given log++/-- setting instead of applying it in the calling context. note that with no args, "$@" will expand to nothing and the var assignments will be applied in the calling context instead of to an executed command. this requires using $(()) syntax instead of let! to set the verbosity.
Aaron Marcuse-Kubitza
02:48 PM Revision 9300: lib/sh/util.sh: logging: inlined PS4 functions into log++/-- because that's the only place they're used and it greatly simplifies the code
Aaron Marcuse-Kubitza
02:34 PM Revision 9299: lib/sh/util.sh: log++/-- (and helpers PS4++/--): added functions for these so they can also be used in other contexts that don't support aliases. note that direct callers of the functions would still need to localize the PS4/$verbosity values using log_local.
Aaron Marcuse-Kubitza
02:08 PM Revision 9298: lib/sh/util.sh: run_args_cmd(): moved test of $? to lib/runscripts/util.run on_exit() since it needs to be performed for all run_args_cmd() functions, not just the default implementation. (this test ensures that no commands are executed if the EXIT trap was encountered due to a parsing/loading error).
Aaron Marcuse-Kubitza
01:54 PM Revision 9297: bugfix: lib/sh/util.sh: fwd(): need to use $top_dir (the runscript's file) instead of "${BASH_SOURCE[1]}" (the caller's file)
Aaron Marcuse-Kubitza
01:38 PM Revision 9296: lib/sh/util.sh: command__set_fds(): cmd_log_fd = log_fd and can_log: don't call set_fds because no redirection is needed. this also avoids the need to increase the log_level of the set_fds call that used to set cmd_log_fd to itself.
Aaron Marcuse-Kubitza
01:30 PM Revision 9295: bugfix: lib/sh/util.sh: command__set_fds(): cmd_log_fd = log_fd: need to increase the log_level *after* can_log is checked, to avoid limiting the command output itself
Aaron Marcuse-Kubitza
01:23 PM Revision 9294: bugfix: lib/sh/util.sh: command__set_fds(): log the command output at the same log_level as the command name (i.e. leave the log_level as-is), to allow nesting verbose commands within one another without the inner commands always getting suppressed
Aaron Marcuse-Kubitza
01:08 PM Revision 9293: lib/sh/util.sh: command__set_fds(): set the src fd outside of the set_fds call, for clarity
Aaron Marcuse-Kubitza
01:02 PM Revision 9292: lib/sh/util.sh: command(): if cmd_log_fd = log_fd, log the filtering of cmd_log_fd at 2 log_levels higher (verbosity >= 3)
Aaron Marcuse-Kubitza
12:57 PM Revision 9291: lib/sh/util.sh: command(): documented that $cmd_log_fd's default (no cmd_log_fd) limits log_fd in case command uses util.sh
Aaron Marcuse-Kubitza
12:54 PM Revision 9290: bugfix: lib/sh/util.sh: command(): need to make $cmd_log_fd a local var when setting its default value
Aaron Marcuse-Kubitza
12:49 PM Revision 9289: lib/sh/util.sh: logging: moved indent for call tree to its own section, so it isn't confused with the log_level
Aaron Marcuse-Kubitza
12:40 PM Revision 9288: lib/sh/util.sh: logging: log_level definition: added instructions to view a message's log_level (count the # of + signs before it in the output)
Aaron Marcuse-Kubitza
12:39 PM Revision 9287: lib/sh/util.sh: logging: log_level definition: replaced command with message because not just commands can be logged
Aaron Marcuse-Kubitza
12:38 PM Revision 9286: lib/sh/util.sh: logging: log_level definition: emphasized that the log_level starts with *1*, not 0 (0 has a special meaning, which refers to unfiltered messages)
Aaron Marcuse-Kubitza
12:34 PM Revision 9285: lib/sh/util.sh: logging: log_level definition: documented that for commands that use can_log(), the log_level starts with 1. for unfiltered commands, the log_level is 0 (i.e. still output at verbosity=0).
Aaron Marcuse-Kubitza
12:29 PM Revision 9284: lib/sh/util.sh: logging: added definition that the log_level is the minimum verbosity needed to display a command
Aaron Marcuse-Kubitza
12:22 PM Revision 9283: lib/sh/util.sh: set_global_fds(): documented that the logging output for this is visible at verbosity >= 3
Aaron Marcuse-Kubitza
12:21 PM Revision 9282: lib/sh/util.sh: set_global_fds(): moved stdlog fd setup to separate setup_log_fd() function, which runs at a much higher log_level (5 instead of 3). because the stdlog fd setup statements *themselves* are logged to fd 2 (because fd 30 is not yet open), you normally don't want to display these, if you are using fd 2 for errors only and fd 30 for logging.
Aaron Marcuse-Kubitza
12:07 PM Revision 9281: lib/sh/util.sh: command(): $cmd_log_fd: if not set, assume it's a script using util.sh (i.e. with log fd=$log_fd)
Aaron Marcuse-Kubitza
12:00 PM Revision 9280: lib/sh/util.sh: set_global_fds(): also open stdlog logging stream on fd 30, and use this in the logging functions by setting $log_fd
Aaron Marcuse-Kubitza
11:51 AM Revision 9279: lib/sh/util.sh: moved streams section before external commands echoing because command() uses set_fds
Aaron Marcuse-Kubitza
11:50 AM Revision 9278: lib/sh/util.sh: moved echo_eval() to general command echoing section so it can be used by other echoing commands
Aaron Marcuse-Kubitza
11:45 AM Revision 9277: lib/sh/util.sh: don't load new aliases after defining echo_stdout because it's no longer an alias
Aaron Marcuse-Kubitza
11:44 AM Revision 9276: lib/sh/util.sh: load new aliases after defining echo_func
Aaron Marcuse-Kubitza
11:36 AM Revision 9275: lib/sh/util.sh: .(): use "$FUNCNAME" like cd()
Aaron Marcuse-Kubitza
11:36 AM Revision 9274: lib/sh/util.sh: .(): wrapped lines in the same way as cd()
Aaron Marcuse-Kubitza
11:34 AM Revision 9273: lib/sh/util.sh: moved functions echoing section before other types of echoing so they can use echo_func()
Aaron Marcuse-Kubitza
11:28 AM Revision 9272: lib/sh/util.sh: $verbosity: decreased default value to 1 so that normally, only external commands are echoed. this avoids cluttering up the output with the function call tree, which is really only needed for debugging. it also mirrors make's default verbosity (i.e. print external commands and subdir recursions, but not make's internal actions).
Aaron Marcuse-Kubitza
11:25 AM Revision 9271: *{.sh,run}: put doc comment on next line when possible, so that the function name always comes before it
Aaron Marcuse-Kubitza
11:22 AM Revision 9270: lib/sh/util.sh: put usage message at end of line when possible, so that the function name always comes before it
Aaron Marcuse-Kubitza
11:14 AM Revision 9269: bugfix: lib/sh/util.sh: $top_script: need to realpath this before doing any cd so this points to the right place. set and use $top_script_abs for this purpose.
Aaron Marcuse-Kubitza
10:34 AM Revision 9268: lib/runscripts/util.run: added setting of $top_file, $top_filename to set_paths()
Aaron Marcuse-Kubitza
10:25 AM Revision 9267: lib/runscripts/util.run: set $top_file, $top_filename in a function (set_paths__util_run())
Aaron Marcuse-Kubitza
10:21 AM Revision 9266: lib/sh/util.sh: cd(): re-run set_paths() to update $top_script, $top_dir for the new current dir
Aaron Marcuse-Kubitza
10:13 AM Revision 9265: lib/sh/util.sh: added set_paths(), which sets $top_script, $top_dir
Aaron Marcuse-Kubitza
10:10 AM Revision 9264: lib/sh/util.sh: moved $top_script, $top_dir before internal commands echoing so they can be reset in cd()
Aaron Marcuse-Kubitza
10:09 AM Revision 9263: lib/sh/util.sh: moved vars echoing before internal commands echoing so that echo_vars can be used in it
Aaron Marcuse-Kubitza
09:56 AM Revision 9262: lib/sh/util.sh: use set_fds() instead of `echo_eval exec` for clarity
Aaron Marcuse-Kubitza
09:53 AM Revision 9261: lib/sh/util.sh: set_fd(): allow setting multiple fds at once, and changed the function name accordingly
Aaron Marcuse-Kubitza
09:50 AM Revision 9260: lib/sh/util.sh: set_fd(): moved noclobber functionality to fd_set_default() since that is only used by fd_set_default() (whose purpose has become just to run set_fd without clobbering an existing dest fd)
Aaron Marcuse-Kubitza
09:40 AM Revision 9259: lib/sh/util.sh: shadow_fd(): renamed to fd_set_default() for clarity (since this can't just be used for shadowing stdin/stdout/stderr)
Aaron Marcuse-Kubitza
09:38 AM Revision 9258: lib/sh/util.sh: set_fd()/shadow_fd(): take the redirect expr as an arg instead of a kw_param because it's the only param
Aaron Marcuse-Kubitza
09:33 AM Revision 9257: lib/sh/util.sh: set_fd()/shadow_fd(): take just a redirect expr containing the entire dest[<>]src expr, and parse the dest out from it
Aaron Marcuse-Kubitza
09:29 AM Revision 9256: lib/sh/util.sh: set_fd()/shadow_fd(): updated usage and kw_params
Aaron Marcuse-Kubitza
09:29 AM Revision 9255: lib/sh/util.sh: set_fd()/shadow_fd(): updated usage and kw_params
Aaron Marcuse-Kubitza
09:24 AM Revision 9254: lib/sh/util.sh: set_fd()/shadow_fd(): take just a redirect source expr containing the <> and the fd, to allow the caller to specify these in the shell redirection syntax. this also allows using a file (such as /dev/null) as the redirect source.
Aaron Marcuse-Kubitza
09:11 AM Revision 9253: lib/sh/util.sh: added cd() wrapper that logs itself and indents (indent is permanent within subshell cd was used in)
Aaron Marcuse-Kubitza
09:04 AM Revision 9252: lib/sh/util.sh: added caller_indent alias
Aaron Marcuse-Kubitza
08:45 AM Revision 9251: lib/sh/util.sh: log_err(): changed symbol to # instead of * so that it stands out better at the outmost indent level and doesn't blend in with the + that will be prefixing other messages
Aaron Marcuse-Kubitza
08:40 AM Revision 9250: lib/sh/util.sh: log_info(): changed symbol to : so that it blends in better with the | lines (looking like a dotted segment of the line), but still stands out (because of the dots)
Aaron Marcuse-Kubitza
08:32 AM Revision 9249: *{.sh,run}: don't use aliases unless it's actually necessary (e.g. to define a var in the calling function, or to alias-expand the next word)
Aaron Marcuse-Kubitza
08:13 AM Revision 9248: lib/runscripts/util.run: $top_file: also match dir-level runscripts by removing /run suffix
Aaron Marcuse-Kubitza
08:13 AM Revision 9247: lib/sh/util.sh: shadow_fd(): take a dest fd instead of a prefix to prepend to the src fd, so that the full name of the dest fd is in the function call. this makes it much clearer what fd will be created from a shadow_fd() call.
Aaron Marcuse-Kubitza
07:59 AM Revision 9246: lib/sh/util.sh: rel_path(): also remove base_dir when path exactly = to it. this involves adding and removing an extra trailing /.
Aaron Marcuse-Kubitza
07:56 AM Revision 9245: bugfix: lib/sh/util.sh: rel_path(): ensure non-empty by using . if empty
Aaron Marcuse-Kubitza
07:55 AM Revision 9244: lib/sh/util.sh: split canon_rel_path() into itself and rel_path() helper for clarity
Aaron Marcuse-Kubitza
07:47 AM Revision 9243: lib/sh/local.sh: $root_dir: run canon_rel_path on it
Aaron Marcuse-Kubitza
07:37 AM Revision 9242: lib/sh/util.sh: log_custom(): use keyword-args syntax instead of local vars
Aaron Marcuse-Kubitza
07:36 AM Revision 9241: lib/sh/util.sh: log_custom(): replace all non-whitespace chars in PS4 with $symbol instead of just the first one, so there aren't mixed $symbols and PS4 chars
Aaron Marcuse-Kubitza
07:28 AM Revision 9240: lib/sh/util.sh: moved paths section after logging so it can use logging functions
Aaron Marcuse-Kubitza
06:10 AM Revision 9239: *{.sh,run}: echo all global vars (except those defined before echo_vars is available)
Aaron Marcuse-Kubitza
05:58 AM Revision 9238: web/links/index.htm.run: use the runscript-common $top_file, which is equivalent to $linked_file
Aaron Marcuse-Kubitza
05:55 AM Revision 9237: *{.sh,run}: use new local_export instead of local...export. this still requires calling echo_vars manually, because local_export is an alias (and must be, to define the var in the caller's context).
Aaron Marcuse-Kubitza
05:54 AM Revision 9236: lib/sh/util.sh: added local_export alias, which combines the effects of local and export
Aaron Marcuse-Kubitza
05:36 AM Revision 9235: lib/sh/util.sh: renamed params() to kw_params() to clarify the type of params it handles
Aaron Marcuse-Kubitza
05:33 AM Revision 9234: lib/sh/util.sh: logging: echo all keyword params by overriding params(). this requires removing manual echo_vars calls.
Aaron Marcuse-Kubitza
05:31 AM Revision 9233: lib/sh/util.sh: moved functions section right after exceptions, so that other sections can use it. in particular, func_override is needed by verbose output to override non-verbose versions of functions.
Aaron Marcuse-Kubitza
05:28 AM Revision 9232: lib/sh/util.sh: only try to echo vars that are set, to avoid `declare -p` "not found" errors
Aaron Marcuse-Kubitza
05:09 AM Revision 9231: *{.sh,run}: use new `params` in all functions that have keyword params, in order to remove their vars from the environment. note that functions that use $? (such as die()) must save it *before* running params, because params will overwrite $?.
Aaron Marcuse-Kubitza
04:28 AM Revision 9230: web/links/index.htm: updated to Firefox bookmarks. added bash and MySQL-related bookmarks, including MySQL doc subfolder.
Aaron Marcuse-Kubitza
04:11 AM Revision 9229: lib/sh/util.sh: params(): added doc comment
Aaron Marcuse-Kubitza
04:10 AM Revision 9228: lib/sh/util.sh: added params(), which removes keyword-param-only vars from the environment
Aaron Marcuse-Kubitza
03:50 AM Revision 9227: lib/sh/util.sh: added unexport() (note that `declare +x` won't work because it defines the var if it isn't set)
Aaron Marcuse-Kubitza
03:39 AM Revision 9226: lib/sh/util.sh: moved vars section to beginning so other sections can use it (it does not contain logging statements, so it doesn't need to be after verbose output)
Aaron Marcuse-Kubitza
03:34 AM Revision 9225: bugfix: lib/sh/util.sh: command(): limit cmd_log_fd directly instead of limiting log_fd (usually fd 2) and then redirecting cmd_log_fd to it. this ensures that stderr is not limited unless it's actually used as the cmd_log_fd, so that a non-logging stderr will still reach the user properly regardless of whether err_fd is set to something other than 2. (previously, without an err_fd at 22, closing fd 2 at the beginning of the set of redirections would leave no fd pointing to a place to echo errors to.)
Aaron Marcuse-Kubitza

05/11/2013

11:16 PM Revision 9224: lib/sh/util.sh: disable_logging(): use echo_eval instead of plain eval so the disabling of logging is nevertheless itself logged
Aaron Marcuse-Kubitza
11:11 PM Revision 9223: lib/sh/util.sh: command(): made usage message an end-of-line comment
Aaron Marcuse-Kubitza
11:10 PM Revision 9222: lib/sh/util.sh: inlined limit_log_fd into command() since it's only used once
Aaron Marcuse-Kubitza
10:22 PM Revision 9221: lib/sh/util.sh: log_err(), log_info(): use bolder symbols (*, #) to make the symbol stand out visually from the line of | above and below it
Aaron Marcuse-Kubitza
10:09 PM Revision 9220: lib/sh/util.sh: $verbosity: lowered the default from 3 to 2 because level 3 output (vars, etc.) is too verbose for the default output and is only needed for debugging
Aaron Marcuse-Kubitza
10:06 PM Revision 9219: lib/sh/util.sh: $log_level_indent: added space after | to visually separate the nesting levels and make the output look like an expandable tree (similar to a filesystem browser tree)
Aaron Marcuse-Kubitza
10:04 PM Revision 9218: lib/sh/util.sh: log_custom(): replace every non-whitespace character of $log_indent, and the first character of PS4, with the symbol, to make the symbol stand out on the line instead of the |||... before it
Aaron Marcuse-Kubitza
09:19 PM Revision 9217: lib/sh/util.sh: removed no longer used stdout2log_fd(). set $cmd_log_fd instead.
Aaron Marcuse-Kubitza
09:16 PM Revision 9216: lib/sh/util.sh: set $log_level_indent to | because this visually connects commands at the same indent by effectively connecting their + prefixes with a line of |
Aaron Marcuse-Kubitza
09:14 PM Revision 9215: lib/sh/archives.sh: zip(), unzip(): added missing echo_func
Aaron Marcuse-Kubitza
09:07 PM Revision 9214: lib/sh/util.sh: command(): use just one control var $cmd_log_fd instead of three flags (limit_log_fd, limit_stdout, stderr_is_errors) that indicated various common fd configurations. this is much clearer (you state which fd the common uses as its logging fd), more configurable (the logging fd can be any fd, not just 1 or 2), and more automatic (redirecting fd 2 to err_fd happens automatically if it isn't used for logging).
Aaron Marcuse-Kubitza
08:58 PM Revision 9213: bugfix: lib/sh/local.sh: psql(): need to use limit_stdout=1 instead of stderr_is_errors=1 because logging info is output to *stdout*, not *stderr*, in spite of the >&2 redirection, since the >&2 is applied *after* command()'s logging redirects. note that the >&2 can now be removed, because it will be performed by command().
Aaron Marcuse-Kubitza
08:16 PM Revision 9212: lib/sh/util.sh: added $log_fd and use it instead of 2 where the log_fd is intended
Aaron Marcuse-Kubitza
08:11 PM Revision 9211: lib/sh/util.sh: moved stdout2log_fd() to command verbose output subsection of verbose output since it is primarily intended for use with external commands
Aaron Marcuse-Kubitza
08:05 PM Revision 9210: lib/sh/util.sh: renamed limit_stderr to limit_log_fd, stdout2stderr to stdout2log_fd to clarify their purpose. log_fd is currently assumed to be 2.
Aaron Marcuse-Kubitza
08:00 PM Revision 9209: lib/sh/util.sh: added disable_logging() and use it instead of `exec 2>/dev/null`
Aaron Marcuse-Kubitza
07:54 PM Revision 9208: lib/sh/util.sh: added $err_fd global var and use it instead of the 22 magic value. it starts out at 2 until fd 22 has been set up, so commands that required 22 will now work before set_global_fds() is called.
Aaron Marcuse-Kubitza
07:11 PM Revision 9207: lib/sh/util.sh: shadow_fd(): don't require it to be run with log++ and instead let callers apply log++ if needed. in set_global_fds(), log++ can just be applied once for all the shadow_fd() calls.
Aaron Marcuse-Kubitza
07:09 PM Revision 9206: lib/sh/util.sh: shadow_fd(): instead of applying minor=1 (i.e. an additional log++) to echo_func and then log++ again separately to set_fd, just apply it to the entire function
Aaron Marcuse-Kubitza
07:07 PM Revision 9205: lib/sh/util.sh: set_fd(): don't increase the log_level for this function, and instead let callers increase it if needed
Aaron Marcuse-Kubitza
06:48 PM Revision 9204: lib/sh/util.sh: limit_stderr(): inlined log++ with the can_log call because it should only apply to it. this also makes it obvious that log++ should *not* apply outside of this function. this also enables the contents of this function to be used in an alias.
Aaron Marcuse-Kubitza
06:01 PM Revision 9203: lib/sh/util.sh: command(): use exec instead of `builtin command` to avoid spawning an extra process (since a subshell is already open)
Aaron Marcuse-Kubitza
05:43 PM Revision 9202: bugfix: lib/sh/util.sh: command(): only indent if command was logged
Aaron Marcuse-Kubitza
05:39 PM Revision 9201: bugfix: lib/sh/util.sh: .(): only indent if include statement was logged
Aaron Marcuse-Kubitza
05:29 PM Revision 9200: lib/sh/util.sh: limit_stderr(): moved $stdout2stderr to command() so that limit_stderr()'s only purpose is to log-limit stderr
Aaron Marcuse-Kubitza
05:23 PM Revision 9199: bugfix: lib/sh/util.sh: limit_stderr(): moved $stderr_is_errors default to command() where it's used
Aaron Marcuse-Kubitza
05:20 PM Revision 9198: lib/sh/util.sh: limit_stderr(): do stdout2stderr *after* limit_stderr to avoid having two similar `can_log ... >/dev/null` commands
Aaron Marcuse-Kubitza
05:09 PM Revision 9197: lib/sh/util.sh: echo_cmd(): removed no longer used removal of first builtin or command arg
Aaron Marcuse-Kubitza
05:07 PM Revision 9196: lib/sh/util.sh: inlined echo_run_prep into command() since it's now only used once
Aaron Marcuse-Kubitza
05:06 PM Revision 9195: lib/sh/util.sh: command(): removed no longer used $cmd_echoed flag
Aaron Marcuse-Kubitza
04:57 PM Revision 9194: *{.sh,run}: use new limit_stdout=1 instead of stdout2stderr=1 limit_stderr=1
Aaron Marcuse-Kubitza
04:56 PM Revision 9193: lib/sh/util.sh: command(): added limit_stdout=1 flag which implies stdout2stderr=1 limit_stderr=1
Aaron Marcuse-Kubitza
04:41 PM Revision 9192: lib/sh/util.sh: merged limit_stderr_cmd/limit_stdout_cmd into command(), using flag vars to control what limiting actions it needs to perform. in command invocations, this involves setting the appropriate flag vars instead of using a limit_std*_cmd alias.
Aaron Marcuse-Kubitza
04:37 PM Revision 9191: lib/sh/util.sh: added run_cmd
Aaron Marcuse-Kubitza
04:04 PM Revision 9190: lib/sh/util.sh: limit_stderr(): moved before command echoing because it does not depend on it and is a separate feature
Aaron Marcuse-Kubitza
03:47 PM Revision 9189: lib/sh/util.sh: run_args_cmd(): use $top_script instead of `"$(canon_rel_path "$0")"`
Aaron Marcuse-Kubitza
03:42 PM Revision 9188: lib/sh/util.sh: $top_script: run canon_rel_path on $0
Aaron Marcuse-Kubitza
03:35 PM Revision 9187: lib/sh/util.sh: use `minor=1 echo_func` instead of echo_minor_func because the syntax is simple enough that you don't need a separate alias for it
Aaron Marcuse-Kubitza
03:32 PM Revision 9186: lib/sh/util.sh: echo_func: support using minor=1 to increase the log_level that the function call is echoed at. use this in echo_minor_func.
Aaron Marcuse-Kubitza
03:28 PM Revision 9185: lib/sh/util.sh: echo_*func aliases: use simpler && syntax (with `|| true` to prevent errexit if the condition is false) instead of an if statement. this also allows prefixing the alias with var definitions.
Aaron Marcuse-Kubitza
03:27 PM Revision 9184: lib/sh/util.sh: echo_*func aliases: use simpler && syntax (with `|| true` to prevent errexit if the condition is false) instead of an if statement. this also allows prefixing the alias with var definitions.
Aaron Marcuse-Kubitza
03:18 PM Revision 9183: lib/sh/util.sh: echo_func: only indent if the function call was displayed. this avoids confusing extra whitespace when a function call causes an indent but the function call itself isn't printed.
Aaron Marcuse-Kubitza
03:10 PM Revision 9182: bugfix: lib/sh/util.sh: echo_minor_func: need to include "$@" in echo_func() call
Aaron Marcuse-Kubitza
03:08 PM Revision 9181: lib/sh/util.sh: echo_func(): added usage
Aaron Marcuse-Kubitza
03:07 PM Revision 9180: lib/sh/util.sh: echo_func: added usage
Aaron Marcuse-Kubitza
02:50 PM Revision 9179: lib/sh/util.sh: $log_level_indent: changed to a single space rather than two, to avoid very deep indents as the call tree becomes more deeply nested. due to the + at the beginning of every line, a single space is enough to visualize the structure of the call tree (unlike source code, which requires more whitespace because it doesn't have a call tree structure).
Aaron Marcuse-Kubitza
02:46 PM Revision 9178: lib/sh/util.sh: echo_*func aliases: increase the indent for the duration of the function call. this causes function calls to be displayed in an indented call-tree structure, which makes the verbose output much easier to understand.
Aaron Marcuse-Kubitza
02:43 PM Revision 9177: bugfix: lib/sh/util.sh: recursive aliases (i.e. aliases that call a command of the same name): use "" around the command name so it isn't reexpanded if the function using the alias is copied using copy_func(). (aliases will be reexpanded every time a function is redeclared since they appear unquoted in the function definition.) this is a bug in `declare -f` for which there is no fix, necessitating the "" alias workaround instead.
Aaron Marcuse-Kubitza
02:01 PM Revision 9176: lib/sh/util.sh: moved log-indent-related vars/aliases before they are first used (by log())
Aaron Marcuse-Kubitza
01:54 PM Revision 9175: lib/sh/util.sh: renamed inc_log_level/dec_log_level to shorter log++/log--
Aaron Marcuse-Kubitza
01:54 PM Revision 9174: lib/sh/util.sh: renamed inc_log_level/dec_log_level to shorter log++/log--
Aaron Marcuse-Kubitza
01:51 PM Revision 9173: lib/sh/util.sh: renamed cmd_indent to just indent now that indent is not being used for the PS4 level
Aaron Marcuse-Kubitza
01:47 PM Revision 9172: *{.sh,run}: multi-command aliases: enclose in {} so the alias can be used after ||
Aaron Marcuse-Kubitza
01:44 PM Revision 9171: lib/sh/util.sh: renamed indent->PS4++ , outdent->PS4-- because these relate just to the PS4 length, not to the (whitespace-based) log_indent
Aaron Marcuse-Kubitza
01:38 PM Revision 9170: lib/sh/util.sh: removed no longer used echo_run(). use command() instead.
Aaron Marcuse-Kubitza
01:36 PM Revision 9169: *{.sh,run}: use command instead of deprecated echo_run (don't prepend anything when the command is already aliased with `command`)
Aaron Marcuse-Kubitza
01:29 PM Revision 9168: lib/sh/util.sh: command(): perform echo_run_prep itself instead of requiring echo_run to be added in an alias. echo_run_prep will not be performed if limit_stderr_cmd() has performed it already outside the limit_stderr section. this requires using `builtin command` instead of "command" when you don't want the echoing (such as in include_guard_var()).
Aaron Marcuse-Kubitza
01:07 PM Revision 9167: *{.sh,run}: multi-command aliases: enclose in {} so the alias can be used after ||
Aaron Marcuse-Kubitza
12:56 PM Revision 9166: bugfix: lib/sh/util.sh: .(): also need to echo the . before the included file
Aaron Marcuse-Kubitza
12:48 PM Revision 9165: lib/sh/util.sh: added echo_params alias and use it instead of `log "$*"`
Aaron Marcuse-Kubitza
12:43 PM Revision 9164: lib/sh/util.sh: .(): echo the include at log_level 2 because it's an internal command
Aaron Marcuse-Kubitza
12:34 PM Revision 9163: lib/sh/util.sh: echo_cmd(): also remove `builtin` before the command name
Aaron Marcuse-Kubitza
12:32 PM Revision 9162: lib/sh/util.sh: echo_cmd(): don't remove -- after command, because it is added by the command() function instead and does not appear in the command sent to echo_run()
Aaron Marcuse-Kubitza
12:29 PM Revision 9161: lib/sh/util.sh: command alias: don't alias-expand next word, because the next word should only be interpreted as a command (part of the effect of the `command` builtin)
Aaron Marcuse-Kubitza
12:27 PM Revision 9160: lib/sh/util.sh: external commands: always use command rather than echo_run to indicate that a command is external, because command() ensures that what it invokes is not a shell function, and sets 2>&22 where needed
Aaron Marcuse-Kubitza
12:23 PM Revision 9159: lib/sh/util.sh: .(): use echo_run so that included files are echoed as they are included. note that echo_run uses cmd_indent, so the included files' load-time logging output will be indented according to the nesting level of the include.
Aaron Marcuse-Kubitza
12:20 PM Revision 9158: lib/sh/util.sh: moved .() command echoing > internal commands section
Aaron Marcuse-Kubitza
12:19 PM Revision 9157: lib/sh/util.sh: moved echo_run() to general command echoing section because it can also be used for internal commands
Aaron Marcuse-Kubitza
12:09 PM Revision 9156: removed no longer used inputs/GBIF/MySQL_export. use lib/sh/local.sh mysql_export_local() instead.
Aaron Marcuse-Kubitza
12:07 PM Revision 9155: inputs/GBIF/raw_occurrence_record/run: override table.tsv/make() instead of export_mysql()
Aaron Marcuse-Kubitza
12:06 PM Revision 9154: inputs/GBIF/table.run: table.tsv/make(): use new mysql_export_local instead of ./MySQL_export
Aaron Marcuse-Kubitza
12:05 PM Revision 9153: lib/sh/local.sh: added mysql_export_local()
Aaron Marcuse-Kubitza
12:05 PM Revision 9152: lib/sh/db.sh: added mysql_export(). documented that mysql_export_outfile() supports CSV, but requires the FILE privilege.
Aaron Marcuse-Kubitza
12:02 PM Revision 9151: lib/sh/db.sh: mysql(): use --column-names to ensure the output is formatted appropriately. note that --column-names is the default, but better to specify it to be sure.
Aaron Marcuse-Kubitza
12:01 PM Revision 9150: lib/sh/db.sh: mysql(): output_data mode: use --batch to ensure the output is formatted appropriately. note that --batch is the default when stdin is from a pipe, but better to specify it to be sure.
Aaron Marcuse-Kubitza
11:49 AM Revision 9149: *{.sh,run}: use new limit_stdout_cmd instead of `stdout2stderr=1 limit_stderr_cmd`
Aaron Marcuse-Kubitza
11:40 AM Revision 9148: lib/sh/util.sh: added limit_stdout_cmd alias
Aaron Marcuse-Kubitza
07:25 AM Revision 9147: lib/sh/make.sh: added $rm short var for $remake
Aaron Marcuse-Kubitza
07:22 AM Revision 9146: lib/sh/util.sh: verbosity: removed `declare -i verbosity` because this does not actually ensure that $verbosity is an integer; it only has "arithmetic evaluation [...] done when the variable is assigned to" (`help declare`)
Aaron Marcuse-Kubitza
07:20 AM Revision 9145: lib/sh/util.sh: verbosity: support setting this in alternate short-name var $vb
Aaron Marcuse-Kubitza
07:15 AM Revision 9144: lib/sh/util.sh: override the `.` builtin (a.k.a. `source`) and run canon_rel_path on the included script. this removes .. in the path when it's displayed in bash error messages.
Aaron Marcuse-Kubitza
06:58 AM Revision 9143: lib/sh/util.sh: log_err(): ensure errors are visible by using 2>&22. since this is uses log(), verbosity=1 also needs to be set.
Aaron Marcuse-Kubitza
06:50 AM Revision 9142: lib/sh/util.sh: $explicit_errors_only: explicitly set verbosity=0 to hide startup logging
Aaron Marcuse-Kubitza
06:46 AM Revision 9141: lib/sh/util.sh: limit_stderr(): if stdout goes to stderr (i.e. is logging info), assume that stderr_is_errors
Aaron Marcuse-Kubitza
06:41 AM Revision 9140: lib/sh/util.sh: $explicit_errors_only: documented that this option should only be used for testing the explicit error displaying, as otherwise, important output may be missed
Aaron Marcuse-Kubitza
06:39 AM Revision 9139: lib/sh/util.sh: $explicit_errors_only: added usage
Aaron Marcuse-Kubitza
06:38 AM Revision 9138: bugfix: lib/sh/util.sh: command(): updated $errors_only var name to $explicit_errors_only
Aaron Marcuse-Kubitza
06:27 AM Revision 9137: lib/sh/util.sh: $explicit_errors_only: noted that verbosity=0 displays everything that isn't explicitly hidden
Aaron Marcuse-Kubitza
06:24 AM Revision 9136: lib/sh/local.sh: psql(): use new stderr_is_errors=1 since psql's logging output comes on stdout, so stderr contains only errors. (the caveat of this is that the query output is normally mixed with logging info on stdout, and needs to be separated out to a different file descriptor using --output.)
Aaron Marcuse-Kubitza
06:18 AM Revision 9135: lib/sh/util.sh: added $explicit_errors_only env var which shows only explicitly-displayed errors (those which have been redirected to fd 22). note that most of the time, this has the same effect as `verbosity=0 script...`.
Aaron Marcuse-Kubitza
06:17 AM Revision 9134: lib/sh/util.sh: command(): support ensuring errors are visible by redirecting them to fd 22 (global stderr) when $stderr_is_errors is set
Aaron Marcuse-Kubitza
05:51 AM Revision 9133: lib/sh/util.sh: `command`: always prepend -- to the args list (to ensure that the command name is not interpreted as a `command` option), not just when the command alias is used
Aaron Marcuse-Kubitza
05:40 AM Revision 9132: lib/sh/util.sh: set_global_fds(): increase the log_level so the shadow_fd()/set_fd() function names aren't output at the default verbosity (3), and the redirection commands themselves aren't output at verbosity <= 2. set_global_fds() happens at the beginning of every script that uses util.sh, and is fairly predictable, so it isn't necessary to always echo these commands.
Aaron Marcuse-Kubitza
05:29 AM Revision 9131: lib/sh/local.sh: psql(): use new convention for command-specific alternate stdout
Aaron Marcuse-Kubitza
05:15 AM Revision 9130: lib/sh/make.sh: inline_make: use new convention for command-specific alternate stdin
Aaron Marcuse-Kubitza
05:13 AM Revision 9129: *{.sh,run}: removed extra space between function name and ()
Aaron Marcuse-Kubitza
05:12 AM Revision 9128: lib/sh/util.sh: streams: moved setting of global stdin/stdout/stderr into set_global_fds () function
Aaron Marcuse-Kubitza
05:05 AM Revision 9127: lib/sh/util.sh: streams: allow commands to access global stdin/stdout/stderr using fd 20/21/22. this works even when /dev/tty isn't available.
Aaron Marcuse-Kubitza
05:05 AM Revision 9126: lib/sh/util.sh: streams: added convention that fd 10/11/12 should be used for command-specific alternate stdin/stdout/stderr
Aaron Marcuse-Kubitza
05:04 AM Revision 9125: lib/sh/util.sh: added streams functions fd_exists, set_fd, shadow_fd, and helpers
Aaron Marcuse-Kubitza
05:01 AM Revision 9124: lib/sh/util.sh: added echo_minor_func alias
Aaron Marcuse-Kubitza
05:00 AM Revision 9123: lib/sh/util.sh: added echo_eval()
Aaron Marcuse-Kubitza
04:32 AM Revision 9122: lib/sh/util.sh: renamed require_exists() to require_not_exists() because the command actually checks if the file *doesn't* exist
Aaron Marcuse-Kubitza
02:26 AM Revision 9121: lib/sh/util.sh: echo_vars(): use log() instead of manually echoing the log indent, so that the log indent string ($log_indent$PS4) doesn't have to be maintained in several different places
Aaron Marcuse-Kubitza
02:23 AM Revision 9120: lib/sh/util.sh: logging: visually separate the log_level-based indenting from the external command indenting by using the standard `set -x` prefix (1st char of PS4) for log_level indenting and $log_level_indent (i.e. whitespace) for external command indenting
Aaron Marcuse-Kubitza
02:19 AM Revision 9119: lib/sh/util.sh: logging: separate the log_level-based indenting from the external command indenting so that the log_level-based indenting can use a different prefix. this involves propagating a new var, $log_indent, to invoked commands instead of $PS4, and always prepending $log_indent to $PS4 when printing log messages.
Aaron Marcuse-Kubitza
01:55 AM Revision 9118: lib/sh/util.sh: propagate $log_level_indent to invoked commands
Aaron Marcuse-Kubitza
01:54 AM Revision 9117: lib/sh/util.sh: renamed $log_indent to $log_level_indent to indicate that this is the string added at each level, rather than the entire indent
Aaron Marcuse-Kubitza
01:49 AM Revision 9116: bugfix: lib/sh/util.sh: limit_stderr(): switched back to running inc_log_level inside limit_stderr(), because it should only apply during the function for use by can_log. documented why inc_log_level needs to be inside limit_stderr().
Aaron Marcuse-Kubitza
01:44 AM Revision 9115: bugfix: lib/sh/util.sh: limit_stderr(): stdout2stderr mode: only limit stdout, since stderr may contain error messages (which should always be displayed)
Aaron Marcuse-Kubitza
01:32 AM Revision 9114: bugfix: lib/sh/util.sh: limit_stderr_cmd(): need to load new aliases before it so limit_stderr is expanded
Aaron Marcuse-Kubitza
01:31 AM Revision 9113: bugfix: lib/sh/util.sh: limit_stderr alias: need to inc_log_level before rather than after limit_stderr(), so that can_log uses the right verbosity (previously, inc_log_level was part of limit_stderr() itself, but was moved to the alias because its effects were needed outside the function)
Aaron Marcuse-Kubitza
01:25 AM Revision 9112: lib/sh/util.sh: stdout2stderr(): fixed doc comment after space-before-() search-and-replace
Aaron Marcuse-Kubitza
01:24 AM Revision 9111: lib/sh/util.sh: usage(): use `return` instead of `(exit ...)`
Aaron Marcuse-Kubitza
01:23 AM Revision 9110: lib/sh/util.sh: verbose output: put commands into subsections
Aaron Marcuse-Kubitza
01:10 AM Revision 9109: lib/sh/util.sh: echo_run(): factored prep code out into echo_run_prep alias, which can also be used in commands that mimic it, such as limit_stderr_cmd()
Aaron Marcuse-Kubitza
01:06 AM Revision 9108: bugfix: lib/sh/util.sh: limit_stderr(): moved indent to echo_run prep, because it only applies to commands invoked with echo_run
Aaron Marcuse-Kubitza
01:00 AM Revision 9107: bugfix: lib/sh/util.sh: limit_stderr(): need to apply inc_log_level in the caller so it won't get rolled back at the end of the function
Aaron Marcuse-Kubitza
12:51 AM Revision 9106: lib/sh/db.sh: mysql_export_outfile(): ensure newline between format info ($mysql_load_data_format) and rest of SELECT query
Aaron Marcuse-Kubitza
12:49 AM Revision 9105: lib/sh/db.sh: mysql_export_outfile(): ensure newline between SELECT columns list and INTO OUTFILE
Aaron Marcuse-Kubitza
12:44 AM Revision 9104: lib/sh/util.sh: limit_stderr_cmd(): moved stdout2stderr code to limit_stderr() because it is not specific to running limit_stderr on single commands
Aaron Marcuse-Kubitza
12:33 AM Revision 9103: lib/sh/util.sh: limit_stderr(): fixed doc comment after space-before-() search-and-replace
Aaron Marcuse-Kubitza
12:29 AM Revision 9102: lib/sh/db.sh: added mysql_export_outfile(), which uses SELECT ... INTO OUTFILE
Aaron Marcuse-Kubitza
12:22 AM Revision 9101: lib/sh/db.sh: mysql(): added $output_data option that turns off --verbose and uses echo_stdin instead so that queries are not echoed to stdout along with data. in --verbose mode, use limit_stderr_cmd with stdout2stderr=1 to redirect echoed queries through the logging mechanism.
Aaron Marcuse-Kubitza
12:17 AM Revision 9100: lib/sh/db.sh: mysql_cmd(): removed set_database because this is now performed by each caller before they use $database
Aaron Marcuse-Kubitza
12:15 AM Revision 9099: lib/sh/db.sh: mysql_cmd: get command name by adding $FUNCNAME in an alias instead of using ${FUNCNAME[1]} in the function, so that callers can also call mysql_cmd via a nested function, etc.
Aaron Marcuse-Kubitza
12:00 AM Revision 9098: lib/sh/util.sh: added ensure_nested_func()
Aaron Marcuse-Kubitza

05/10/2013

11:59 PM Revision 9097: lib/sh/util.sh: added contains()
Aaron Marcuse-Kubitza
11:57 PM Revision 9096: lib/sh/util.sh: command alias: also prepend -- to args to prevent a first arg starting with -- from being interpreted as an option to the `command` builtin
Aaron Marcuse-Kubitza
10:16 PM Revision 9095: lib/sh/db.sh: mysql_cmd(): moved setting of database to specific callers (mysql(), mysqldump()) because the syntax to set it differs between mysql* commands (e.g. mysql uses `--database=` while mysqldump uses `--databases ... --tables`)
Aaron Marcuse-Kubitza
10:13 PM Revision 9094: lib/sh/db.sh: added set_database alias, and use it in mysql_cmd()
Aaron Marcuse-Kubitza
10:11 PM Revision 9093: lib/sh/util.sh: import_vars alias and applicable aliases that use it: documented that when used inside another alias 2+ levels deep, it *must* be run inside a function. this is due to a strange bug in bash where $() expressions in 2-level aliases produce a syntax error when the alias is expanded outside a function.
Aaron Marcuse-Kubitza
10:07 PM Revision 9092: bugfix: lib/sh/util.sh: added back missing echo_func alias, which is needed to also include "$@" in the echoed function call
Aaron Marcuse-Kubitza
10:05 PM Revision 9091: lib/sh/util.sh: import_vars alias: use new get_prefix_vars()
Aaron Marcuse-Kubitza
10:04 PM Revision 9090: lib/sh/util.sh: add get_prefix_vars()
Aaron Marcuse-Kubitza
10:03 PM Revision 9089: lib/sh/util.sh: import_vars alias: use declare instead of local so it can be used outside a function
Aaron Marcuse-Kubitza
02:47 PM Revision 9088: lib/sh/db.sh: added mk_select alias and use it in pg_export()
Aaron Marcuse-Kubitza
02:36 PM Revision 9087: lib/sh/db.sh: pg_export(): always include the LIMIT value if specified, rather than only if the caller did not provide a full query
Aaron Marcuse-Kubitza
02:27 PM Revision 9086: lib/sh/db.sh: pg_export(): rtrim $query
Aaron Marcuse-Kubitza
02:22 PM Revision 9085: lib/sh/util.sh: added rtrim()
Aaron Marcuse-Kubitza
01:55 PM Revision 9084: lib/sh/db.sh: pg_export(): for simplicity, always use a SELECT statement as the source. take any source query in \$query without () instead of \$source with (). this will also help make pg_export uniform with MySQL.
Aaron Marcuse-Kubitza
01:34 PM Revision 9083: lib/sh/db.sh: renamed pg_copy_to() to pg_export() because it is also a general-purpose export command, which could have an analogous counterpart for MySQL
Aaron Marcuse-Kubitza
01:23 PM Revision 9082: bugfix: lib/sh/util.sh: moved self alias to beginning, before functions that use it
Aaron Marcuse-Kubitza
12:55 PM Revision 9081: lib/sh/util.sh: added repeat()
Aaron Marcuse-Kubitza
12:47 PM Revision 9080: lib/sh/util.sh: reverse(): use `i > 0` rather than `i >= 1` to match the `i < length` idiom used in forwards loops
Aaron Marcuse-Kubitza
12:43 PM Revision 9079: lib/sh/util.sh: moved strings section before verbose output so its commands can be used by the logging functions
Aaron Marcuse-Kubitza
12:32 PM Revision 9078: lib/sh/util.sh: echo_run() and derivatives (limit_stderr_cmd()): use new `indent` before the invoked command so its own logging messages, if any, are indented
Aaron Marcuse-Kubitza
12:27 PM Revision 9077: lib/sh/util.sh: propagate indent to invoked commands by exporting PS4
Aaron Marcuse-Kubitza
12:24 PM Revision 9076: lib/sh/util.sh: verbose output: added indent/outdent aliases and use them in inc_log_level/dec_log_level
Aaron Marcuse-Kubitza
12:14 PM Revision 9075: *{.sh,run}: replaced extern with command, a shell builtin that does the same thing. this is also part of dash (the Bourne shell on Linux).
Aaron Marcuse-Kubitza
12:08 PM Revision 9074: *{.sh,run}: removed extra space between function name and (), which is apparently not needed even though `help function` includes it. this greatly improves readability, including when function names are pasted into commit messages!
Aaron Marcuse-Kubitza
12:03 PM Revision 9073: lib/sh/util.sh: require_exists (): use die/log_info's new support for info messages using $type
Aaron Marcuse-Kubitza
12:00 PM Revision 9072: lib/sh/util.sh: die (): allow caller to specify a custom log message type (i.e. a suffix for log_* ())
Aaron Marcuse-Kubitza
11:58 AM Revision 9071: lib/sh/util.sh: log_e (): use new log_err
Aaron Marcuse-Kubitza
11:57 AM Revision 9070: lib/sh/util.sh: verbose output: added log_custom (), which modifies the symbol in PS4 to indicate the message type, and log_err ()/log_info (), which set specific symbols
Aaron Marcuse-Kubitza
11:55 AM Revision 9069: lib/sh/util.sh: moved die () into verbose output so it can use logging functions
Aaron Marcuse-Kubitza
11:53 AM Revision 9068: lib/sh/util.sh: moved functions after verbose output so copy_func () can use logging functions in its input checks
Aaron Marcuse-Kubitza
11:41 AM Revision 9067: lib/sh/util.sh: moved log_e () to verbose output section so it can use logging functions
Aaron Marcuse-Kubitza
11:29 AM Revision 9066: lib/sh/db.sh: mysqldump_diffable (): use pipe_delay to ensure that sed is printed after mysqldump
Aaron Marcuse-Kubitza
11:23 AM Revision 9065: lib/sh/util.sh: echo_cmd (): use new log ()
Aaron Marcuse-Kubitza
11:21 AM Revision 9064: lib/sh/util.sh: added log ()
Aaron Marcuse-Kubitza
10:53 AM Revision 9063: lib/sh/util.sh to_file, make.sh check_target_exists: use new require_exists to print message if file skipped because it already exists. this is useful for troubleshooting why certain make commands don't run.
Aaron Marcuse-Kubitza
10:51 AM Revision 9062: lib/sh/util.sh: added require_exists (), used to skip make commands for existing files
Aaron Marcuse-Kubitza
10:33 AM Revision 9061: *{.sh,run}: use `test !` instead of `! test` so that the ! is right next to the operator it's negating
Aaron Marcuse-Kubitza
10:31 AM Revision 9060: /run: geoscrub_input/make (): use new check_fake_target_exists to create the file only if it doesn't exist yet
Aaron Marcuse-Kubitza
10:31 AM Revision 9059: lib/sh/make.sh: added check_fake_target_exists (analogous to check_target_exists), which defers the target existence check until to_file (when the target name will presumably have been resolved to a path)
Aaron Marcuse-Kubitza
10:29 AM Revision 9058: lib/sh/util.sh: to_file (): support only running if the file does not exist by setting $if_not_exists
Aaron Marcuse-Kubitza
10:08 AM Revision 9057: lib/sh/util.sh: limit_stderr_cmd (): run cmd2rel_path explicitly here because echo_run is split apart rather than being run as echo_run ()
Aaron Marcuse-Kubitza
10:06 AM Revision 9056: lib/sh/util.sh: moved echo_cmd, echo_run before commands that use them
Aaron Marcuse-Kubitza
10:02 AM Revision 9055: lib/sh/make.sh: check_target_exists: if remaking, consider target not to exist
Aaron Marcuse-Kubitza
10:01 AM Revision 9054: lib/sh/make.sh: added remaking alias
Aaron Marcuse-Kubitza
09:45 AM Revision 9053: inputs/GBIF/table.run: table.tsv/make (): use echo_run instead of extern so that the command name is canonicalized properly
Aaron Marcuse-Kubitza
09:45 AM Revision 9052: lib/sh/util.sh: echo_run (): use new cmd2rel_path to resolve the command ($1)
Aaron Marcuse-Kubitza
09:44 AM Revision 9051: lib/sh/util.sh: added cmd2rel_path alias, which makes $1 a canon_rel_path if it's a filesystem path. this removes extra .. in the paths of invoked commands.
Aaron Marcuse-Kubitza
09:06 AM Revision 9050: *{.sh,run}: use shorter `test` instead of `test -n` and `test !` instead of `test -z` (http://www.gnu.org/software/bash/manual/bash.html#Bourne-Shell-Builtins > test)
Aaron Marcuse-Kubitza
08:52 AM Revision 9049: lib/sh/util.sh: echo_cmd (): don't include leading extern because it clutters up the output and is implied by the log_level
Aaron Marcuse-Kubitza
08:45 AM Revision 9048: lib/sh/util.sh: to_file (): run echo_func and `echo_vars stdout` to show what file the output is going to, since this information (i.e. redirects) isn't included in the logging output for the command itself
Aaron Marcuse-Kubitza
08:41 AM Revision 9047: lib/runscripts/table.run: remake_VegBIEN_mappings (): added public_schema_exists check, ported from lib/import.sh
Aaron Marcuse-Kubitza
08:40 AM Revision 9046: lib/sh/local.sh: added public_schema_exists (), ported from lib/import.sh
Aaron Marcuse-Kubitza
08:32 AM Revision 9045: *{.sh,run}: use new echo_stdout instead of echo_stdin where applicable, for clarity
Aaron Marcuse-Kubitza
08:31 AM Revision 9044: lib/sh/util.sh: added echo_stdout (currently just an alias of echo_stdin, because they are usable for the same purpose)
Aaron Marcuse-Kubitza
08:27 AM Revision 9043: /run: moved geoscrub_input export into separate geoscrub_input/make () target
Aaron Marcuse-Kubitza
08:24 AM Revision 9042: lib/sh/db.sh: pg_export_table_to_dir_no_header (): use to_file so that the file is autoremoved in case of error
Aaron Marcuse-Kubitza
07:51 AM Revision 9041: lib/sh/util.sh: let (): renamed to let! so that let can still be used to evaluate whether a numeric value is 0 (yes, an ! is allowed at the end of a command name)
Aaron Marcuse-Kubitza
07:44 AM Revision 9040: lib/sh/util.sh: moved canon_rel_path () into separate paths section
Aaron Marcuse-Kubitza
07:39 AM Revision 9039: bugfix: *{.sh,run}: use new func_override for runscript inheritance instead of invoking the overridden function as a (command-line) target of the parent runscript. this ensures that the overridden function is run in the *same* process as the calling function, so that $top_dir keeps the same value and runscript-relative paths continue to work.
Aaron Marcuse-Kubitza
07:30 AM Revision 9038: lib/sh/util.sh: added func_override (), for use in runscript inheritance
Aaron Marcuse-Kubitza
07:03 AM Revision 9037: lib/sh/util.sh: copy_func (): check that $from exists and $to does not exist (i.e. don't clobber existing functions). you can always first `unset -f` the function to get around the no-clobber restriction.
Aaron Marcuse-Kubitza
07:00 AM Revision 9036: lib/sh/util.sh: exceptions: added die ()
Aaron Marcuse-Kubitza
06:44 AM Revision 9035: lib/sh/util.sh: log_e (): restructured as an error handler to put after save_e rather than as a wrapper around the entire command. this allows it to be used with any kind of expression (such as boolean expressions with !), not just single commands.
Aaron Marcuse-Kubitza
06:35 AM Revision 9034: lib/sh/util.sh: log_e (): don't export $e to the calling context, since the caller can just use ` || { save_e; ...; }` if they need $e
Aaron Marcuse-Kubitza
06:31 AM Revision 9033: lib/sh/util.sh: calls to log_e: don't rely on log_e setting $e
Aaron Marcuse-Kubitza
06:30 AM Revision 9032: lib/sh/util.sh: added save_e, which now sets $e just locally
Aaron Marcuse-Kubitza
06:28 AM Revision 9031: lib/sh/util.sh: save_e: renamed to export_e because $e overwrites any previous value in the calling context
Aaron Marcuse-Kubitza
06:26 AM Revision 9030: lib/sh/util.sh: removed no longer used save_e_cmd
Aaron Marcuse-Kubitza
06:24 AM Revision 9029: lib/sh/util.sh: try (): use simpler save_e instead of save_e_cmd
Aaron Marcuse-Kubitza
06:19 AM Revision 9028: bugfix: lib/sh/util.sh: bool2int (): need to use try instead of save_e_cmd because save_e_cmd rethrows the error, which should instead just be stored in $e. this bug was not found in testing because bool2int was only used in $(), which errexit does not apply to.
Aaron Marcuse-Kubitza
06:15 AM Revision 9027: lib/sh/util.sh: exceptions: added save_e, now an alias for e=$?. added save_e/rethrow usage.
Aaron Marcuse-Kubitza
06:11 AM Revision 9026: lib/sh/util.sh: renamed save_e () to save_e_cmd () since it actually runs a command, in addition to saving $?
Aaron Marcuse-Kubitza
06:08 AM Revision 9025: lib/sh/util.sh: log_e (): rewrote to avoid using save_e, which will be repurposed
Aaron Marcuse-Kubitza
05:50 AM Revision 9024: lib/sh/util.sh: added func_exists ()
Aaron Marcuse-Kubitza
05:50 AM Revision 9023: lib/sh/util.sh: moved copy_func () after exceptions so it can use save_e/rethrow
Aaron Marcuse-Kubitza
05:44 AM Revision 9022: lib/sh/util.sh: added copy_func ()
Aaron Marcuse-Kubitza
05:39 AM Revision 9021: lib/sh/util.sh: save_e (): usage: added rethrow example
Aaron Marcuse-Kubitza
05:26 AM Revision 9020: bugfix: lib/sh/util.sh: bool2int (): need to load new aliases before it so that save_e will be expanded
Aaron Marcuse-Kubitza
05:24 AM Revision 9019: *{.sh,run}: put functions on one line where possible (and where they are not expected to expand)
Aaron Marcuse-Kubitza
05:19 AM Revision 9018: lib/sh/util.sh: added separate include guard around the include guard utils so they don't have to be redefined on every include of util.sh
Aaron Marcuse-Kubitza
05:06 AM Revision 9017: *{.sh,run}: added extra line before new sections to visually separate them. lib/sh/util.sh: added missing section headers.
Aaron Marcuse-Kubitza
05:01 AM Revision 9016: lib/sh/util.sh: split make utils out into separate make.sh
Aaron Marcuse-Kubitza
04:38 AM Revision 9015: lib/sh/util.sh: split archive (zip) utils out into separate archives.sh
Aaron Marcuse-Kubitza
04:24 AM Revision 9014: lib/sh/util.sh: split databases utils out into separate db.sh
Aaron Marcuse-Kubitza
04:05 AM Revision 9013: moved lib/*.sh to sh/ subdir so it's easier to find the .sh files among all the other lib/ files
Aaron Marcuse-Kubitza
03:32 AM Revision 9012: lib/util.sh: removed no longer used limit_stderr_extern. use `limit_stderr_cmd extern` instead.
Aaron Marcuse-Kubitza
03:29 AM Revision 9011: lib/util.sh: specify limit_stderr_cmd just for the (few) commands that need it, rather than for all commands, so that commands that use stderr to print important error messages don't have those error messages hidden when the verbosity is too low. (error messages should always be displayed, regardless of the verbosity.)
Aaron Marcuse-Kubitza
03:13 AM Revision 9010: lib/util.sh: limit_stderr_cmd (): only echo the command if it starts with echo_run. this requires adding echo_run before commands that use limit_stderr_cmd, such as limit_stderr_extern.
Aaron Marcuse-Kubitza
03:07 AM Revision 9009: lib/util.sh: limit_stderr_cmd (): remove echo_run from the command to run so that the command name isn't echoed twice
Aaron Marcuse-Kubitza
02:56 AM Revision 9008: lib/util.sh: limit_stderr_cmd: alias-expand command after it
Aaron Marcuse-Kubitza
02:53 AM Revision 9007: lib/util.sh: $verbosity: ensure it's an integer using `declare -i`
Aaron Marcuse-Kubitza
02:45 AM Revision 9006: lib/util.sh: grouped set verbosity statements together and commented them
Aaron Marcuse-Kubitza
02:43 AM Revision 9005: bugfix: lib/util.sh: verbose output: $verbosity defaults to $verbose (boolean) converted to integer. the previous set-default of $verbosity to $verbose has been removed because it came after `: "${verbosity=3}"` and thus didn't have an effect.
Aaron Marcuse-Kubitza
02:40 AM Revision 9004: lib/util.sh: added bool2int ()
Aaron Marcuse-Kubitza
02:32 AM Revision 9003: *{.sh,run}: use new isset
Aaron Marcuse-Kubitza
02:31 AM Revision 9002: lib/util.sh: added isset ()
Aaron Marcuse-Kubitza
02:28 AM Revision 9001: *{.sh,run}: use ${var+isset} instead of ${var+t} for clarity
Aaron Marcuse-Kubitza
02:22 AM Revision 9000: lib/util.sh: propagate the verbosity to invoked commands by exporting it
Aaron Marcuse-Kubitza
02:20 AM Revision 8999: lib/util.sh: run all commands verbosely by default, not just runscripts. this ensures verbose output for invoked commands like inputs/GBIF/MySQL_export.
Aaron Marcuse-Kubitza
02:15 AM Revision 8998: lib/util.sh: echo_stdin (): use new pipe_delay
Aaron Marcuse-Kubitza
02:12 AM Revision 8997: lib/util.sh: added pipe_delay (used as `cmd1 | { pipe_delay; cmd2; }`)
Aaron Marcuse-Kubitza
02:06 AM Revision 8996: lib/util.sh: limit_stderr_cmd (): echo the command like echo_run so that callers don't have to separately call echo_run. this reduces clutter of the nested aliases, ensures that the command is always echoed *outside* of the inner stderr-limiter (which has a different log_level), and avoids echoing "limit_stderr_cmd" itself as part of the command name.
Aaron Marcuse-Kubitza
01:47 AM Revision 8995: lib/util.sh: limit_stderr (): increase the log_level so that stderr of verbose commands can be turned off separately from the names of the commands themselves. it will now usually have log_level 2, indicating output that is useful primarily for debugging (this is the same as for shell function calls).
Aaron Marcuse-Kubitza
01:39 AM Revision 8994: lib/util.sh: renamed log_stderr* to limit_stderr* to reflect that stderr is limited (i.e. controlled) rather than logged
Aaron Marcuse-Kubitza
01:36 AM Revision 8993: lib/util.sh: echo_run_extern: renamed to log_stderr_extern since controlling stderr is its primary function
Aaron Marcuse-Kubitza
01:25 AM Revision 8992: inputs/GBIF/MySQL_export: only include WHERE clause if $filter is set. support configuring LIMIT/OFFSET.
Aaron Marcuse-Kubitza
01:00 AM Revision 8991: inputs/GBIF/table.run: table.tsv/make (): use new to_target to auto-delete $target on error
Aaron Marcuse-Kubitza
12:59 AM Revision 8990: inputs/GBIF/table.run: table.tsv/make (): use new check_target_exists
Aaron Marcuse-Kubitza
12:58 AM Revision 8989: lib/util.sh: make: added check_target_exists
Aaron Marcuse-Kubitza
12:42 AM Revision 8988: lib/util.sh: moved $top_file* to runscripts/util.run because they only apply to runscripts
Aaron Marcuse-Kubitza
12:36 AM Revision 8987: lib/util.sh: make: added to_target, which uses to_file on $target
Aaron Marcuse-Kubitza
12:35 AM Revision 8986: lib/util.sh: added to_file (), which auto-removes a command's output file on error (like make's .DELETE_ON_ERROR)
Aaron Marcuse-Kubitza
12:31 AM Revision 8985: lib/util.sh: echo_run_extern, extern: fixed/added comments to indicate that echo_run_extern echoes and controls stderr of *an* external command, while the extern alias does this for *all* external commands
Aaron Marcuse-Kubitza
12:28 AM Revision 8984: lib/util.sh: auto-echo common external commands, such as rm
Aaron Marcuse-Kubitza
12:24 AM Revision 8983: lib/util.sh: echo_run_extern, extern aliases: alias-expand next word (the command) by adding trailing space to alias def
Aaron Marcuse-Kubitza
12:21 AM Revision 8982: bugfix: lib/util.sh: exceptions: log_e: must include `declare e` in the alias and not when save_e is called, so that $e is a local var *of the caller*. this bug did not appear in testing because the save_e alias, which re-scopes $e within log_e (), was not expanded inside log_e () (since new aliases were not loaded between save_e and log_e ()).
Aaron Marcuse-Kubitza
12:12 AM Revision 8981: lib/util.sh: exceptions: added log_e (), which prints a "command exited with error" message (like make) when applicable
Aaron Marcuse-Kubitza
12:09 AM Revision 8980: lib/util.sh: exceptions: try_ (): renamed to try () and use the function keyword to distinguish it from the alias
Aaron Marcuse-Kubitza
12:06 AM Revision 8979: lib/util.sh: exceptions: try_ (): use new save_e
Aaron Marcuse-Kubitza
12:05 AM Revision 8978: lib/util.sh: exceptions: added save_e
Aaron Marcuse-Kubitza
12:01 AM Revision 8977: lib/util.sh: exceptions: end_try* aliases: use new rethrow*
Aaron Marcuse-Kubitza
 

Also available in: Atom