Project

General

Profile

Activity

From 04/18/2013 to 05/17/2013

05/17/2013

06:00 PM Revision 9459: bugfix: mappings/VegCore-VegBIEN.csv: place.geovalid: added missing /1 after _alt
Aaron Marcuse-Kubitza
05:55 PM Revision 9458: bugfix: lib/sql.py: parse_exception(): typed_name_re: added back matching of names without "", since these are used by some error messages (ones that contain () after the function name)
Aaron Marcuse-Kubitza
05:41 PM Revision 9457: bugfix: lib/sql.py: parse_exception(): typed_name_re: need to allow " within the matched name, since there are now "" around the entire identifer that was passed to Postgres, which may itself include " . always require "" around the matched name, to ensure that the whole name is matched by .+? e.g. when followed by () for a function call. the version of Postgres we currently use apparently no longer has error messages without the "", so we don't need a separate regexp for quoted and unquoted names.
Aaron Marcuse-Kubitza
03:43 PM Revision 9456: lib/sh/db.sh: mysql_import(): automatically ensure the table is empty (i.e. using truncate()), unless append=1 is specified. extra calls to truncate() now that this happens automatically have also been removed.
Aaron Marcuse-Kubitza
01:13 PM Revision 9455: bin/map: by_col: ensure verbosity is at least 2 in live mode (using new ints.set_min() instead of max() for clarity). documented that live column-based import MUST be run with verbosity 2+ (3 preferred) to provide debugging information for often-complex errors. without this, debugging is effectively impossible.
Aaron Marcuse-Kubitza
01:08 PM Revision 9454: added lib/ints.py with renamings of max()->set_*min*(), min()->set_*max*() for easier understandability of the set-ceiling/set-floor use cases of min()/max()
Aaron Marcuse-Kubitza
12:57 PM Revision 9453: bin/map: Set default verbosity: by_col: documented that showing all queries is primarily to assist *debugging*, not profiling
Aaron Marcuse-Kubitza
11:59 AM Revision 9452: lib/sh/util.sh: logging: named it `log++`
Aaron Marcuse-Kubitza
11:59 AM Revision 9451: lib/sh/util.sh: logging: verbosities: level 0: documented that log++ also suppresses external command output for full support of cron jobs
Aaron Marcuse-Kubitza
11:57 AM Revision 9450: lib/sh/util.sh: logging: documented `make` equivalents of the various verbosities, where available. (many of the verbosities, such as level 1, are sorely needed in make to avoid excessive output.)
# verbosities (and `make` equivalents):
# 0: just print errors. useful for cron jobs.
# vs. make: equivalent to --...
Aaron Marcuse-Kubitza
04:03 AM Revision 9449: lib/sh/util.sh: die_e(): benign errors: increase log_level so that a benign non-zero exit status will only be displayed at debug verbosities (2+) (it is confusing otherwise)
Aaron Marcuse-Kubitza
03:36 AM Revision 9448: lib/sh/util.sh: try(): always run the command with benign_error=1 so that any die_e() doesn't prematurely indicate that a particular exit status was an error
Aaron Marcuse-Kubitza
03:34 AM Revision 9447: lib/sh/util.sh: die_e(): support benign errors using $benign_error flag that should be logged as info messages instead of errors
Aaron Marcuse-Kubitza
03:30 AM Revision 9446: lib/sh/util.sh: die(): documented that msg can't use $() (because it would reset $?)
Aaron Marcuse-Kubitza
03:19 AM Revision 9445: inputs/bien_web/observation/VegBIEN.csv, unmapped_terms.csv: regenerated
Aaron Marcuse-Kubitza
03:01 AM Revision 9444: lib/sh/util.sh: command(): 2>&$err_fd: add to _redirs *after* echoing command so it isn't echoed at the end of every command (since this redirection is frequently applied)
Aaron Marcuse-Kubitza
02:55 AM Revision 9443: lib/sh/util.sh: sed: use case statement instead of test to determine flag letter, to easily allow matching multiple `uname` OSes or adding additional flag letters
Aaron Marcuse-Kubitza
02:46 AM Revision 9442: lib/sh/util.sh: die(): documented that its msg can use $?, because it has not yet been overridden by another command
Aaron Marcuse-Kubitza
02:45 AM Revision 9441: lib/sh/util.sh: die_e(): use die(), which performs the necessary save_e/rethrow. this requires using $? instead of $e for the exit status, because $e has not yet been set.
Aaron Marcuse-Kubitza
02:42 AM Revision 9440: lib/sh/util.sh: inlined log_e() into die_e() because that's the only place it's used
Aaron Marcuse-Kubitza
02:37 AM Revision 9439: lib/sh/util.sh: command(): print "command exited with error" message using new die_e() if command returns false. this requires removing manual die_e()/log_e() calls elsewhere.
Aaron Marcuse-Kubitza
02:34 AM Revision 9438: lib/sh/util.sh: command(): moved increase of indent inside () so that error-handling statements after () will use the outer log_level
Aaron Marcuse-Kubitza
02:31 AM Revision 9437: lib/sh/util.sh: added die_e(), which logs that a command exited with an error
Aaron Marcuse-Kubitza
02:18 AM Revision 9436: lib/sh/util.sh: command(): determine redirections before echoing the command so they can be logged along with the command, instead of as separate exec statements. (these had a higher log_level to avoid cluttering the output with `exec` lines, which usually suppressed the redirections completely.) inline the command__set_fds() nested func so the redirections are all in one place.
Aaron Marcuse-Kubitza
01:54 AM Revision 9435: lib/sh/util.sh: use simpler `if can_log; then indent; fi` instead of `can_log && indent || true`. however, the `&& indent || true` syntax is still required in aliases such as echo_func which need to allow prefixing the command with a wrapper command or kw param assignments.
Aaron Marcuse-Kubitza

05/16/2013

09:28 PM Revision 9434: inputs/GBIF/raw_occurrence_record/run: dynamically generate herbaria_filter.csv_ from herbaria.ih in new target herbaria_filter.csv_/make()
Aaron Marcuse-Kubitza
09:27 PM Revision 9433: inputs/GBIF/raw_occurrence_record/run: store the herbaria filter in a MySQL table loaded from a CSV instead of getting it from a hardcoded list of IN (...) values
Aaron Marcuse-Kubitza
09:24 PM Revision 9432: lib/sh/db.sh: added truncate()
Aaron Marcuse-Kubitza
09:23 PM Revision 9431: lib/sh/make.sh: set_make_vars: set $target_stem
Aaron Marcuse-Kubitza
08:49 PM Revision 9430: lib/sh/db.sh: added mysql_import()
Aaron Marcuse-Kubitza
07:02 PM Revision 9429: lib/sh/db.sh: removed no longer used mk_esc_name()
Aaron Marcuse-Kubitza
07:01 PM Revision 9428: lib/runscripts/table.run: don't mk_esc_name schema, table because these will be mk_esc_name'd by functions that use them
Aaron Marcuse-Kubitza
06:55 PM Revision 9427: lib/sh/local.sh: psql(): use $schema_esc, $table_esc instead of just putting $schema, $table in ""
Aaron Marcuse-Kubitza
06:48 PM Revision 9426: lib/sh/db.sh: mk_esc_name_alias(): don't overwrite an already-defined $*_esc, to allow the user to provide an already-escaped value (such as a schema-qualified table) directly
Aaron Marcuse-Kubitza
06:38 PM Revision 9425: lib/sh/util.sh: rtrim(): increase the log_level of sed to 4+ instead of 2+ because it is usually run as part of a var assignment, and should therefore have a lower log_level than echo_vars
Aaron Marcuse-Kubitza
06:32 PM Revision 9424: lib/sh/db.sh: mk_esc_name_alias(): echo_vars the *_esc var when it's set
Aaron Marcuse-Kubitza
06:31 PM Revision 9423: lib/sh/db.sh: added mk_esc_name_alias() and use it to create mk_schema_esc, mk_table_esc
Aaron Marcuse-Kubitza
05:55 PM Revision 9422: lib/sh/db.sh: mysql(): run with --local-infile=1
Aaron Marcuse-Kubitza
05:48 PM Revision 9421: bugfix: lib/sh/db.sh: log_sql(): use can_log() instead because the verbosity now gets decremented as the log_level increases, so the threshold to compare to is 0 instead of 2
Aaron Marcuse-Kubitza
05:46 PM Revision 9420: lib/sh/util.sh: added set_default()
Aaron Marcuse-Kubitza
05:45 PM Revision 9419: lib/sh/util.sh: rtrim(): run at higher log_level so that sed command is not normally echoed
Aaron Marcuse-Kubitza
04:40 PM Revision 9418: inputs/GBIF/raw_occurrence_record/run: renamed herbaria.sql to herbaria.data.sql so it wouldn't be added to svn by `make inputs/GBIF/raw_occurrence_record/add` or `make inputs/add`
Aaron Marcuse-Kubitza
04:38 PM Revision 9417: inputs/input.Makefile: $(svnFiles): also exclude *.data.sql, which should never be in svn
Aaron Marcuse-Kubitza
04:27 PM Revision 9416: schemas/vegbien.sql: cultivated_family_locations: documented that table is from sftp://nimoy.nceas.ucsb.edu/home/bien/bien2_scripts/geoscrub/cultivated/cult_by_taxon/flag_by_taxa.inc (i.e. not generated by a function)
Aaron Marcuse-Kubitza
04:15 PM Revision 9415: schemas/vegbien.sql: place.geovalid: added latLongDomainValid to the values to _and together
Aaron Marcuse-Kubitza
04:09 PM Revision 9414: schemas/vegbien.sql: place.geovalid: require it to be NOT NULL so that it's always a 2-valued boolean (but default it to false since it's not a required field)
Aaron Marcuse-Kubitza
04:06 PM Revision 9413: mappings/VegCore-VegBIEN.csv: place.geovalid: use false instead of NULL
Aaron Marcuse-Kubitza
03:46 PM Revision 9412: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): exclude deleted rows (i.e. where the deleted timestamp is non-NULL)
Aaron Marcuse-Kubitza
03:42 PM Revision 9411: inputs/GBIF/raw_occurrence_record/header.csv: regenerated using ./run. since the table is reimported as a CSV, it uses bin/csv2db, which prepends an additional row_num column.
Aaron Marcuse-Kubitza
03:09 PM Revision 9410: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): remove explicit cols list to include all cols. the file size of the generated table.tsv will increase by ~3x, but should remain reasonably-sized compared to our available disk space.
Aaron Marcuse-Kubitza
03:04 PM Revision 9409: bugfix: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): need \ line continuation after vars so they only apply to the command rather than being set as global vars
Aaron Marcuse-Kubitza
03:02 PM Revision 9408: bugfix: lib/runscripts/table.run: load_data(): use new $verbosity_min instead of running `verbosity_min` so that the command name logging is not output with the new verbosity
Aaron Marcuse-Kubitza
02:59 PM Revision 9407: lib/sh/util.sh: added $verbosity_min to set a `verbosity_min` value *after* the command name, etc. has been logged, so that the logging itself is not output with the new verbosity
Aaron Marcuse-Kubitza
02:38 PM Revision 9406: schemas/vegbien.sql: range_modeling_input: include only plants (i.e. rows with higher_plant_group IS NOT NULL)
Aaron Marcuse-Kubitza
02:36 PM Revision 9405: schemas/vegbien.sql: range_modeling_input: added higher_plant_group, for use in restricting rows to plants
Aaron Marcuse-Kubitza
02:24 PM Revision 9404: inputs/.geoscrub/geoscrub_output/map.csv: *validity: added definitions of the numeric codes from _src/README.TXT
Aaron Marcuse-Kubitza
01:11 PM Revision 9403: added planning/workflow/validation/GeoDistKM.sql.txt
Aaron Marcuse-Kubitza
12:42 PM Revision 9402: planning/goals/BIEN3_derived_data_products.docx: updated to most recent version from Brad's e-mail on 2013-4-16
Aaron Marcuse-Kubitza
11:18 AM Revision 9401: /README.TXT: Full database import: before running screen: added `unset TMOUT` because TMOUT (autologout) causes screen to exit even with background processes active
Aaron Marcuse-Kubitza
11:17 AM Revision 9400: /README.TXT: Maintenance: added things to put in your .profile on a live machine (e.g. vegbiendev). in particular, you *MUST NOT* have a TMOUT (autologout) set, because this causes screen to exit even if background processes (e.g. from column-based import) are running
Aaron Marcuse-Kubitza
11:15 AM Revision 9399: /README.TXT: Maintenance: added things to put in your .profile on a live machine (e.g. vegbiendev). in particular, you *MUST NOT* have a TMOUT (autologout) set, because this causes screen to exit even if background processes (e.g. from column-based import) are running
Aaron Marcuse-Kubitza
06:15 AM Revision 9398: bugfix: lib/runscripts/table.run: load_data(): need to ensure the verbosity is at least 3 because the install logs require verbose output. (3 is the default for the installer, but is overridden by the runscripts, which instead set the default to 1.)
Aaron Marcuse-Kubitza
06:13 AM Revision 9397: lib/sh/util.sh: logging: added verbosity_min()
Aaron Marcuse-Kubitza
06:00 AM Revision 9396: lib/sh/util.sh: logging: added verbosity_int() and use it instead of `round_down "$verbosity"`
Aaron Marcuse-Kubitza
02:08 AM Revision 9395: bugfix: lib/sh/util.sh: log+ alias: don't expand next word because it's not a cmd
Aaron Marcuse-Kubitza
01:59 AM Revision 9394: lib/runscripts/table.run: $schema, $table: log the `cd` used to calculate the value at log_level 3 instead of 1 (note that the cd() function call for this will be logged at log_level 5)
Aaron Marcuse-Kubitza
01:51 AM Revision 9393: lib/sh/util.sh: cd(): trace-log the cd() function call itself (at log_level 3) in addition to the cd builtin call
Aaron Marcuse-Kubitza

05/15/2013

11:28 PM Revision 9392: lib/runscripts/table.run: import(): added step to load the data into the staging table before postprocessing it
Aaron Marcuse-Kubitza
11:27 PM Revision 9391: inputs/GBIF/raw_occurrence_record/run: moved table.tsv.md5/make() and invocation of it to inputs/GBIF/table.run because it's general to all tables (which would all use table.tsv for this datasource). use $target_filename in calling table.tsv.md5/make from table.tsv/make.
Aaron Marcuse-Kubitza
11:21 PM Revision 9390: bugfix: lib/sql.py: parse_exception(): typed_name_re: need to ensure that full name is matched rather than just first character
Aaron Marcuse-Kubitza
10:26 PM Revision 9389: web/links/index.htm: updated to Firefox bookmarks. Mountain Lion upgrade: added Python psycopg2, Python OrderedDict, X11.
Aaron Marcuse-Kubitza
07:51 PM Revision 9388: lib/runscripts/table.run: table_make(): run canon_rel_path on the datasrc dir so it's displayed as the direct path to it, without ..
Aaron Marcuse-Kubitza
07:47 PM Revision 9387: lib/runscripts/table.run: table_make(): moved comment about "${@/#/$table/}" to right after the line it describes
Aaron Marcuse-Kubitza
07:44 PM Revision 9386: lib/runscripts/table.run: input_make(): renamed to table_make() to make it clear that the target names are relative to the table subdir itself, not the datasrc dir. it was previously called input_make because it used inputs/input.Makefile directly, but now will use any Makefile in the datasrc dir.
Aaron Marcuse-Kubitza
06:44 PM Revision 9385: added inputs/GBIF/Makefile, which links to ../input.Makefile, to allow running make directly in the datasrc dir (i.e. without --makefile=.../input.Makefile). this is *required* by the runscripts.
Aaron Marcuse-Kubitza
03:34 PM Revision 9384: inputs/GBIF/raw_occurrence_record/run: table.tsv.md5/make(): don't add extra .md5 extension to $target_filename because it already has the extension as part of the target name (now that this command is run in its own make target rather than in table.tsv/make())
Aaron Marcuse-Kubitza
03:30 PM Revision 9383: lib/sql_gen.py: import OrderedDict from collections instead of ordereddict for Mac 10.8 Mountain Lion upgrade
Aaron Marcuse-Kubitza
02:29 PM Revision 9382: lib/sh/util.sh: logging: renamed $log_level_indent to $log_indent_step to avoid confusion with the log_level, which is a different kind of indent (using + signs instead of |s)
Aaron Marcuse-Kubitza
02:26 PM Revision 9381: lib/sh/util.sh: logging: also export PS4, because it follows verbosity and therefore also needs to be propagated to invoked commands
Aaron Marcuse-Kubitza
02:21 PM Revision 9380: lib/sh/util.sh: can_log(): support decimal verbosities using round_down()
Aaron Marcuse-Kubitza
02:21 PM Revision 9379: lib/sh/util.sh: always use echo_export instead of export, even when the verbosity at load time would suppress output, because the verbosity may actually increase during the script due to log-- calls, etc., and vars should then still be echoed as expected
Aaron Marcuse-Kubitza
02:17 PM Revision 9378: lib/sh/util.sh: log+: inlined PS4_prefix_n alias because there is now room for it
Aaron Marcuse-Kubitza
02:16 PM Revision 9377: lib/sh/util.sh: log+: $verbosity: support decimal verbosities (but not decimal log_levels) by using new float+int
Aaron Marcuse-Kubitza
02:13 PM Revision 9376: lib/sh/util.sh: removed no longer used log-. use log+ with a negative argument instead.
Aaron Marcuse-Kubitza
02:12 PM Revision 9375: bugfix: lib/sh/util.sh: log+: PS4 when $1 < 0: need to negate $1 because now it's a negative number
Aaron Marcuse-Kubitza
02:06 PM Revision 9374: lib/sh/util.sh: log--: use log+ with -1 instead of log- so we don't need a separate log- function
Aaron Marcuse-Kubitza
02:04 PM Revision 9373: lib/sh/util.sh: logging: log+: support negative log_level adjustments. log-: use log+ with the negative of its argument
Aaron Marcuse-Kubitza
01:48 PM Revision 9372: web/links/index.htm: updated to Firefox bookmarks. sorted MySQL and PostgreSQL sections.
Aaron Marcuse-Kubitza
01:42 PM Revision 9371: web/links/index.htm: updated to Firefox bookmarks
Aaron Marcuse-Kubitza
01:39 PM Revision 9370: web/links/index.htm: Mac 10.8 Mountain Lion > PostgreSQL: appended tab to name to disambiguate it from the general PostgreSQL section
Aaron Marcuse-Kubitza
01:33 PM Revision 9369: bugfix: web/links/index.htm.run: prepend __ to the HTML anchors of bookmarks toolbar links so they don't shadow/override folders of the same name: also renamed the corresponding self-anchor hyperlinks
Aaron Marcuse-Kubitza
01:32 PM Revision 9368: bugfix: web/links/index.htm.run: prepend __ to the HTML anchors of bookmarks toolbar links so they don't shadow/override folders of the same name: need to replace *all* occurrences (using /g option to sed) to include both HTML anchors on the line
Aaron Marcuse-Kubitza
01:26 PM Revision 9367: web/links/index.htm.run: prepend __ to the HTML anchors of bookmarks toolbar links so they don't shadow/override folders of the same name
Aaron Marcuse-Kubitza
01:23 PM Revision 9366: lib/sh/util.sh: sed: changed it to an alias so it will also be expanded when passed to an external command (like in_place) that can only run an executable, not a shell function (this occurs as long as the external command is defined as an alias which ends in space, to alias-expand the next word). added associated $sed_cmd var for cases when there is no alias wrapper around the external command, and the literal alias body must be used instead.
Aaron Marcuse-Kubitza
11:02 AM Revision 9365: lib/sh/util.sh: log+/-(): setting verbosity: added space around operators to support negative numbers
Aaron Marcuse-Kubitza
10:55 AM Revision 9364: lib/sh/util.sh: added float functions (esp. float+int()) for dealing with decimal verbosities used by sql.py and column-based import
Aaron Marcuse-Kubitza
10:53 AM Revision 9363: inputs/GBIF/raw_occurrence_record/run: inputs/GBIF/raw_occurrence_record/run: added check_target_exists so you know why make skipped the file (for other, non-silent targets, it would also avoid make's verbose output when the file exists)
Aaron Marcuse-Kubitza
10:38 AM Revision 9362: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): moved making of table.tsv.md5 to separate function
Aaron Marcuse-Kubitza
10:30 AM Revision 9361: lib/sh/util.sh: $log_fd: indicated that this is only *initially* stderr (however, the new port will just use stderr if it's not redirected separately)
Aaron Marcuse-Kubitza
10:28 AM Revision 9360: lib/sh/util.sh: verbosities: 3: added that this includes values of kw params
Aaron Marcuse-Kubitza
10:25 AM Revision 9359: lib/sh/util.sh: logging: added description of the verbosities available, including what each one does and what it's useful for:
# verbosities:
# 0: just print errors. useful for cron jobs.
# 1: also external commands run. useful for running at t...
Aaron Marcuse-Kubitza
10:11 AM Revision 9358: lib/sh/util.sh: command__set_fds(): echo the >& line at a higher log_level, because this information (i.e. which fd is used by the command for logging) is primarily for debugging and should not normally be printed
Aaron Marcuse-Kubitza
10:00 AM Revision 9357: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): also add md5 sum for table.tsv
Aaron Marcuse-Kubitza
09:50 AM Revision 9356: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added back filter kw args, which had gotten deleted in a commit without update (although actually, svn should not allow a commit without update, so the working copy may have gotten corrupted)
Aaron Marcuse-Kubitza
09:27 AM Revision 9355: lib/runscripts/table.run: input_make(): added $silent flag which turns on make's --silent option
Aaron Marcuse-Kubitza
09:25 AM Revision 9354: lib/sh/make.sh: set_make_vars: usage: added "use $target" to indicate that vars are made available by this alias
Aaron Marcuse-Kubitza
09:24 AM Revision 9353: lib/sh/make.sh: inline_make(): take the script stdin from caller-provided stdin and get make's stdin from global stdin, so that the caller can just use <<'EOF' rather than having to include a specific fd before the <<
Aaron Marcuse-Kubitza
08:46 AM Revision 9352: lib/runscripts/table.run: input_make(): documented that it requires a Makefile in the datasrc dir containing `include ../input.Makefile
Aaron Marcuse-Kubitza
08:41 AM Revision 9351: lib/runscripts/table.run: input_make(): added comment explaining that "${@/#/$table/}" replaces the empty str at the beginning of str (/#) with $table/
Aaron Marcuse-Kubitza
08:38 AM Revision 9350: lib/runscripts/table.run: put functions on one line where possible
Aaron Marcuse-Kubitza
08:35 AM Revision 9349: lib/runscripts/table.run: input_make(): use the local make() function instead of external make directly, because it sets $cmd_log_fd appropriately to ensure that all the echoed make commands get properly logged to stdlog/stderr (stdlog is fd 30 when it's redirected to a file)
Aaron Marcuse-Kubitza
08:22 AM Revision 9348: inputs/GBIF/raw_occurrence_record/run table.tsv/make() and functions used by it: added usage comments for cmd line usage, caller usage, and declaring function usage
Aaron Marcuse-Kubitza
07:29 AM Revision 9347: web/links/index.htm: updated to Firefox bookmarks
Aaron Marcuse-Kubitza
07:27 AM Revision 9346: web/links/index.htm: updated to Firefox bookmarks
Aaron Marcuse-Kubitza
07:00 AM Revision 9345: web/links/index.htm: removed no longer needed - at the beginning of every folder's description
Aaron Marcuse-Kubitza
07:00 AM Revision 9344: lib/Firefox_bookmarks.reformat.csv: label page's description: don't do this for folders (i.e. descriptions preceded by an <H3> tag) because their descriptions are always author-added rather than from a web page. this avoids needing to add a - at the beginning of every folder's description.
Aaron Marcuse-Kubitza

05/14/2013

09:43 PM Revision 9343: web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: put fixes subdirs in order.
Aaron Marcuse-Kubitza
09:42 PM Revision 9342: web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: put fixes subdirs in order.
Aaron Marcuse-Kubitza
08:56 PM Revision 9341: web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: added fixes for Apache ~/Sites dirs, Apache PHP
Aaron Marcuse-Kubitza
05:55 PM Revision 9340: web/links/index.htm: updated to Firefox bookmarks. added bookmarks for upgrading Mac OS X to 10.8 Mountain Lion. (WARNING: DO NOT upgrade unless you are prepared to fix several programs broken by the upgrade: svn, PostgreSQL. instructions are in the corresponding bookmark subdirs. these programs will be COMPLETELY UNAVAILABLE until they are manually fixed!)
Aaron Marcuse-Kubitza

05/13/2013

07:43 AM Revision 9339: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): cols: also include scientific_name, which is preferable as a TNRS input because it also contains lower ranks
Aaron Marcuse-Kubitza
07:40 AM Revision 9338: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): cols: also include id, institution_code, collection_code, catalogue_number
Aaron Marcuse-Kubitza
07:38 AM Revision 9337: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added filter for institution_codes in herbaria.ih (in PostgreSQL)
Aaron Marcuse-Kubitza
07:19 AM Revision 9336: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added column subset (from http://vegpath.org/twiki/bin/view/Main/ConfCall20130509#subsetting_strategy > include)
Aaron Marcuse-Kubitza
07:14 AM Revision 9335: lib/sh/db.sh: mk_select: constructed queries: in ${var:+if_true} syntax, put the newline at the end of the if_true value instead of the beginning, so that each ${var:+if_true} expression starts at the beginning of a line
Aaron Marcuse-Kubitza
07:10 AM Revision 9334: lib/sh/db.sh: mk_select: constructed queries: support custom columns list using $cols
Aaron Marcuse-Kubitza
07:07 AM Revision 9333: lib/sh/db.sh: mk_select: constructed queries: support WHERE clause using $filter
Aaron Marcuse-Kubitza
06:52 AM Revision 9332: lib/sh/util.sh: echo_stdin(): don' increase the log_level, because the input being sent to the command (which is usually a set of interpreted commands itself) is necessary to fully know what action is being performed
Aaron Marcuse-Kubitza
06:30 AM Revision 9331: bugfix: lib/sh/local.sh: add a manual errexit for $() exprs by embedding them in just a var assignment (*without* local or declare), whose exit status will then equal the of the $(). a `|| return` also needs to be added because errexit does not work on assignment statements. this commit adds them for func_loc(), echo_func(), canon_rel_path(), set_paths(), save_cache, cached realpath(), local.sh global vars
Aaron Marcuse-Kubitza
06:14 AM Revision 9330: lib/sh/util.sh: shell-variable-based caching: usage: updated alias names
Aaron Marcuse-Kubitza
05:33 AM Revision 9329: web/links/index.htm: updated to Firefox bookmarks. regenerating also removed extra <a name=...> tags that were added when running index.htm.run on an already-processed index.htm.
Aaron Marcuse-Kubitza
05:19 AM Revision 9328: bugfix: lib/sh/util.sh: `set -o`: need a -o before *every* option to set
Aaron Marcuse-Kubitza
05:10 AM Revision 9327: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' names: hyperlink the added anchors as clickable paragraph marks (like Redmine), which take you to the HTML anchor. this is analogous to the clickable folder names which take you to *their* anchors.
Aaron Marcuse-Kubitza
04:52 AM Revision 9326: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' *names*, in addition to their URLs
Aaron Marcuse-Kubitza
04:47 AM Revision 9325: bugfix: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' URLs: use .*? instead of .* to match the contents of the <A> tag before the HREF
Aaron Marcuse-Kubitza
04:45 AM Revision 9324: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links using the URL itself as the anchor. these can be used to link to the comments attached *to* a bookmark in the bookmarks page, rather than to the bookmark's destination.
Aaron Marcuse-Kubitza
04:43 AM Revision 9323: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links using the URL itself as the anchor. these can be used to link to the comments attached *to* a bookmark in the bookmarks page, rather than to the bookmark's destination.
Aaron Marcuse-Kubitza
04:35 AM Revision 9322: web/links/index.htm: updated to Firefox bookmarks
Aaron Marcuse-Kubitza
04:24 AM Revision 9321: lib/sh/util.sh: `set -o`: added pipefail option, to ensure that exit statuses (esp. for errexit) also work with pipelines (a|b)
Aaron Marcuse-Kubitza
04:13 AM Revision 9320: lib/sh/util.sh: use new log+ with a numeric arg instead of multiple calls to log++
Aaron Marcuse-Kubitza
04:06 AM Revision 9319: lib/sh/util.sh: use new log+ with a numeric arg instead of multiple calls to log++. use the command form of log-- (`log-- echo_func`) to counter the normal log++ performed by echo_func, for cases when the function name is descriptive and should be output at the same log_level as the commands it runs.
Aaron Marcuse-Kubitza

05/12/2013

08:33 PM Revision 9318: *{.sh,run}: removed extra space between function name and ()
Aaron Marcuse-Kubitza
08:13 PM Revision 9317: lib/sh/util.sh: echo_func(): take the FUNCNAME as an argument (auto-added in the echo_func alias) instead of getting it from the FUNCNAME array (which would have produced an inaccurate value if another function call (such as log++) intervened between the caller and echo_func())
Aaron Marcuse-Kubitza
08:04 PM Revision 9316: lib/sh/util.sh: echo_func(): display where the function was declared (using new func_loc()) instead of where echo_func() was called from. this is more intuitive when debugging, becaues the line # is where the function starts. it also helps remove the dependency on the FUNCNAME/BASH_* arrays, which would produce an inaccurate value if another function call (such as log++) intervened between the caller and echo_func().
Aaron Marcuse-Kubitza
07:57 PM Revision 9315: lib/sh/util.sh: added func_loc(), which gets where a function was declared in the format file:line, and helper alias set_func_loc
Aaron Marcuse-Kubitza
07:09 PM Revision 9314: lib/sh/util.sh: str2varname(): use bash's internal ${var//glob/repl} syntax, which supports character classes ([a-z], etc.), instead of sed (which is slower because it's an external command and uses regexps)
Aaron Marcuse-Kubitza
07:00 PM Revision 9313: lib/sh/util.sh: added support for caching realpath() using the new shell-variable-based caching. this can be enabled via $realpath_cache and defaults to off because it's currently slower than without. note that the cache needs to be cleared in cd() because relative paths will become invalid.
Aaron Marcuse-Kubitza
06:48 PM Revision 9312: lib/sh/util.sh: added str2varname() and use it in include_guard_var()
Aaron Marcuse-Kubitza
06:47 PM Revision 9311: lib/sh/util.sh: added shell-variable-based caching functions
Aaron Marcuse-Kubitza
05:08 PM Revision 9310: lib/sh/util.sh: canon_rel_path(): use $PWD instead of the slower $(pwd -P) now that $PWD has symlinks expanded
Aaron Marcuse-Kubitza
05:06 PM Revision 9309: lib/sh/util.sh: expand symlinks in $PWD: moved it before canon_rel_path(), which will require it
Aaron Marcuse-Kubitza
05:03 PM Revision 9308: lib/sh/util.sh: expand symlinks in $PWD using `cd -P .` so it matches the output of realpath
Aaron Marcuse-Kubitza
05:01 PM Revision 9307: lib/sh/util.sh: cd(): documented what -P option does and why it's used
Aaron Marcuse-Kubitza
04:55 PM Revision 9306: lib/sh/util.sh: cd(): always run cd with -P (expand symlinks) so that $PWD will contain a canonical path which matches the output of `readlink -f`. however, don't display the added -P in the logging output because it distracts from the directory being changed to.
Aaron Marcuse-Kubitza
04:29 PM Revision 9305: lib/sh/util.sh: logging: added log+/-, which take a variable log_level step, and use them in log++/--
Aaron Marcuse-Kubitza
04:16 PM Revision 9304: lib/sh/util.sh: repeat(): for simplicity and speed, just append to a local string var (and echo the result at the end) instead of using printf
Aaron Marcuse-Kubitza
04:12 PM Revision 9303: bugfix: lib/sh/util.sh: repeat(): need to use %s instead of %q (escaped string) in printf
Aaron Marcuse-Kubitza
03:59 PM Revision 9302: lib/sh/util.sh: logging: log++/--: comment about the different uses can be performed with one function: clarified that *without a cmd*, "$@" expands to nothing and assignments are applied to the caller
Aaron Marcuse-Kubitza
03:00 PM Revision 9301: lib/sh/util.sh: log++/--: support running a command with the given log++/-- setting instead of applying it in the calling context. note that with no args, "$@" will expand to nothing and the var assignments will be applied in the calling context instead of to an executed command. this requires using $(()) syntax instead of let! to set the verbosity.
Aaron Marcuse-Kubitza
02:48 PM Revision 9300: lib/sh/util.sh: logging: inlined PS4 functions into log++/-- because that's the only place they're used and it greatly simplifies the code
Aaron Marcuse-Kubitza
02:34 PM Revision 9299: lib/sh/util.sh: log++/-- (and helpers PS4++/--): added functions for these so they can also be used in other contexts that don't support aliases. note that direct callers of the functions would still need to localize the PS4/$verbosity values using log_local.
Aaron Marcuse-Kubitza
02:08 PM Revision 9298: lib/sh/util.sh: run_args_cmd(): moved test of $? to lib/runscripts/util.run on_exit() since it needs to be performed for all run_args_cmd() functions, not just the default implementation. (this test ensures that no commands are executed if the EXIT trap was encountered due to a parsing/loading error).
Aaron Marcuse-Kubitza
01:54 PM Revision 9297: bugfix: lib/sh/util.sh: fwd(): need to use $top_dir (the runscript's file) instead of "${BASH_SOURCE[1]}" (the caller's file)
Aaron Marcuse-Kubitza
01:38 PM Revision 9296: lib/sh/util.sh: command__set_fds(): cmd_log_fd = log_fd and can_log: don't call set_fds because no redirection is needed. this also avoids the need to increase the log_level of the set_fds call that used to set cmd_log_fd to itself.
Aaron Marcuse-Kubitza
01:30 PM Revision 9295: bugfix: lib/sh/util.sh: command__set_fds(): cmd_log_fd = log_fd: need to increase the log_level *after* can_log is checked, to avoid limiting the command output itself
Aaron Marcuse-Kubitza
01:23 PM Revision 9294: bugfix: lib/sh/util.sh: command__set_fds(): log the command output at the same log_level as the command name (i.e. leave the log_level as-is), to allow nesting verbose commands within one another without the inner commands always getting suppressed
Aaron Marcuse-Kubitza
01:08 PM Revision 9293: lib/sh/util.sh: command__set_fds(): set the src fd outside of the set_fds call, for clarity
Aaron Marcuse-Kubitza
01:02 PM Revision 9292: lib/sh/util.sh: command(): if cmd_log_fd = log_fd, log the filtering of cmd_log_fd at 2 log_levels higher (verbosity >= 3)
Aaron Marcuse-Kubitza
12:57 PM Revision 9291: lib/sh/util.sh: command(): documented that $cmd_log_fd's default (no cmd_log_fd) limits log_fd in case command uses util.sh
Aaron Marcuse-Kubitza
12:54 PM Revision 9290: bugfix: lib/sh/util.sh: command(): need to make $cmd_log_fd a local var when setting its default value
Aaron Marcuse-Kubitza
12:49 PM Revision 9289: lib/sh/util.sh: logging: moved indent for call tree to its own section, so it isn't confused with the log_level
Aaron Marcuse-Kubitza
12:40 PM Revision 9288: lib/sh/util.sh: logging: log_level definition: added instructions to view a message's log_level (count the # of + signs before it in the output)
Aaron Marcuse-Kubitza
12:39 PM Revision 9287: lib/sh/util.sh: logging: log_level definition: replaced command with message because not just commands can be logged
Aaron Marcuse-Kubitza
12:38 PM Revision 9286: lib/sh/util.sh: logging: log_level definition: emphasized that the log_level starts with *1*, not 0 (0 has a special meaning, which refers to unfiltered messages)
Aaron Marcuse-Kubitza
12:34 PM Revision 9285: lib/sh/util.sh: logging: log_level definition: documented that for commands that use can_log(), the log_level starts with 1. for unfiltered commands, the log_level is 0 (i.e. still output at verbosity=0).
Aaron Marcuse-Kubitza
12:29 PM Revision 9284: lib/sh/util.sh: logging: added definition that the log_level is the minimum verbosity needed to display a command
Aaron Marcuse-Kubitza
12:22 PM Revision 9283: lib/sh/util.sh: set_global_fds(): documented that the logging output for this is visible at verbosity >= 3
Aaron Marcuse-Kubitza
12:21 PM Revision 9282: lib/sh/util.sh: set_global_fds(): moved stdlog fd setup to separate setup_log_fd() function, which runs at a much higher log_level (5 instead of 3). because the stdlog fd setup statements *themselves* are logged to fd 2 (because fd 30 is not yet open), you normally don't want to display these, if you are using fd 2 for errors only and fd 30 for logging.
Aaron Marcuse-Kubitza
12:07 PM Revision 9281: lib/sh/util.sh: command(): $cmd_log_fd: if not set, assume it's a script using util.sh (i.e. with log fd=$log_fd)
Aaron Marcuse-Kubitza
12:00 PM Revision 9280: lib/sh/util.sh: set_global_fds(): also open stdlog logging stream on fd 30, and use this in the logging functions by setting $log_fd
Aaron Marcuse-Kubitza
11:51 AM Revision 9279: lib/sh/util.sh: moved streams section before external commands echoing because command() uses set_fds
Aaron Marcuse-Kubitza
11:50 AM Revision 9278: lib/sh/util.sh: moved echo_eval() to general command echoing section so it can be used by other echoing commands
Aaron Marcuse-Kubitza
11:45 AM Revision 9277: lib/sh/util.sh: don't load new aliases after defining echo_stdout because it's no longer an alias
Aaron Marcuse-Kubitza
11:44 AM Revision 9276: lib/sh/util.sh: load new aliases after defining echo_func
Aaron Marcuse-Kubitza
11:36 AM Revision 9275: lib/sh/util.sh: .(): use "$FUNCNAME" like cd()
Aaron Marcuse-Kubitza
11:36 AM Revision 9274: lib/sh/util.sh: .(): wrapped lines in the same way as cd()
Aaron Marcuse-Kubitza
11:34 AM Revision 9273: lib/sh/util.sh: moved functions echoing section before other types of echoing so they can use echo_func()
Aaron Marcuse-Kubitza
11:28 AM Revision 9272: lib/sh/util.sh: $verbosity: decreased default value to 1 so that normally, only external commands are echoed. this avoids cluttering up the output with the function call tree, which is really only needed for debugging. it also mirrors make's default verbosity (i.e. print external commands and subdir recursions, but not make's internal actions).
Aaron Marcuse-Kubitza
11:25 AM Revision 9271: *{.sh,run}: put doc comment on next line when possible, so that the function name always comes before it
Aaron Marcuse-Kubitza
11:22 AM Revision 9270: lib/sh/util.sh: put usage message at end of line when possible, so that the function name always comes before it
Aaron Marcuse-Kubitza
11:14 AM Revision 9269: bugfix: lib/sh/util.sh: $top_script: need to realpath this before doing any cd so this points to the right place. set and use $top_script_abs for this purpose.
Aaron Marcuse-Kubitza
10:34 AM Revision 9268: lib/runscripts/util.run: added setting of $top_file, $top_filename to set_paths()
Aaron Marcuse-Kubitza
10:25 AM Revision 9267: lib/runscripts/util.run: set $top_file, $top_filename in a function (set_paths__util_run())
Aaron Marcuse-Kubitza
10:21 AM Revision 9266: lib/sh/util.sh: cd(): re-run set_paths() to update $top_script, $top_dir for the new current dir
Aaron Marcuse-Kubitza
10:13 AM Revision 9265: lib/sh/util.sh: added set_paths(), which sets $top_script, $top_dir
Aaron Marcuse-Kubitza
10:10 AM Revision 9264: lib/sh/util.sh: moved $top_script, $top_dir before internal commands echoing so they can be reset in cd()
Aaron Marcuse-Kubitza
10:09 AM Revision 9263: lib/sh/util.sh: moved vars echoing before internal commands echoing so that echo_vars can be used in it
Aaron Marcuse-Kubitza
09:56 AM Revision 9262: lib/sh/util.sh: use set_fds() instead of `echo_eval exec` for clarity
Aaron Marcuse-Kubitza
09:53 AM Revision 9261: lib/sh/util.sh: set_fd(): allow setting multiple fds at once, and changed the function name accordingly
Aaron Marcuse-Kubitza
09:50 AM Revision 9260: lib/sh/util.sh: set_fd(): moved noclobber functionality to fd_set_default() since that is only used by fd_set_default() (whose purpose has become just to run set_fd without clobbering an existing dest fd)
Aaron Marcuse-Kubitza
09:40 AM Revision 9259: lib/sh/util.sh: shadow_fd(): renamed to fd_set_default() for clarity (since this can't just be used for shadowing stdin/stdout/stderr)
Aaron Marcuse-Kubitza
09:38 AM Revision 9258: lib/sh/util.sh: set_fd()/shadow_fd(): take the redirect expr as an arg instead of a kw_param because it's the only param
Aaron Marcuse-Kubitza
09:33 AM Revision 9257: lib/sh/util.sh: set_fd()/shadow_fd(): take just a redirect expr containing the entire dest[<>]src expr, and parse the dest out from it
Aaron Marcuse-Kubitza
09:29 AM Revision 9256: lib/sh/util.sh: set_fd()/shadow_fd(): updated usage and kw_params
Aaron Marcuse-Kubitza
09:29 AM Revision 9255: lib/sh/util.sh: set_fd()/shadow_fd(): updated usage and kw_params
Aaron Marcuse-Kubitza
09:24 AM Revision 9254: lib/sh/util.sh: set_fd()/shadow_fd(): take just a redirect source expr containing the <> and the fd, to allow the caller to specify these in the shell redirection syntax. this also allows using a file (such as /dev/null) as the redirect source.
Aaron Marcuse-Kubitza
09:11 AM Revision 9253: lib/sh/util.sh: added cd() wrapper that logs itself and indents (indent is permanent within subshell cd was used in)
Aaron Marcuse-Kubitza
09:04 AM Revision 9252: lib/sh/util.sh: added caller_indent alias
Aaron Marcuse-Kubitza
08:45 AM Revision 9251: lib/sh/util.sh: log_err(): changed symbol to # instead of * so that it stands out better at the outmost indent level and doesn't blend in with the + that will be prefixing other messages
Aaron Marcuse-Kubitza
08:40 AM Revision 9250: lib/sh/util.sh: log_info(): changed symbol to : so that it blends in better with the | lines (looking like a dotted segment of the line), but still stands out (because of the dots)
Aaron Marcuse-Kubitza
08:32 AM Revision 9249: *{.sh,run}: don't use aliases unless it's actually necessary (e.g. to define a var in the calling function, or to alias-expand the next word)
Aaron Marcuse-Kubitza
08:13 AM Revision 9248: lib/runscripts/util.run: $top_file: also match dir-level runscripts by removing /run suffix
Aaron Marcuse-Kubitza
08:13 AM Revision 9247: lib/sh/util.sh: shadow_fd(): take a dest fd instead of a prefix to prepend to the src fd, so that the full name of the dest fd is in the function call. this makes it much clearer what fd will be created from a shadow_fd() call.
Aaron Marcuse-Kubitza
07:59 AM Revision 9246: lib/sh/util.sh: rel_path(): also remove base_dir when path exactly = to it. this involves adding and removing an extra trailing /.
Aaron Marcuse-Kubitza
07:56 AM Revision 9245: bugfix: lib/sh/util.sh: rel_path(): ensure non-empty by using . if empty
Aaron Marcuse-Kubitza
07:55 AM Revision 9244: lib/sh/util.sh: split canon_rel_path() into itself and rel_path() helper for clarity
Aaron Marcuse-Kubitza
07:47 AM Revision 9243: lib/sh/local.sh: $root_dir: run canon_rel_path on it
Aaron Marcuse-Kubitza
07:37 AM Revision 9242: lib/sh/util.sh: log_custom(): use keyword-args syntax instead of local vars
Aaron Marcuse-Kubitza
07:36 AM Revision 9241: lib/sh/util.sh: log_custom(): replace all non-whitespace chars in PS4 with $symbol instead of just the first one, so there aren't mixed $symbols and PS4 chars
Aaron Marcuse-Kubitza
07:28 AM Revision 9240: lib/sh/util.sh: moved paths section after logging so it can use logging functions
Aaron Marcuse-Kubitza
06:10 AM Revision 9239: *{.sh,run}: echo all global vars (except those defined before echo_vars is available)
Aaron Marcuse-Kubitza
05:58 AM Revision 9238: web/links/index.htm.run: use the runscript-common $top_file, which is equivalent to $linked_file
Aaron Marcuse-Kubitza
05:55 AM Revision 9237: *{.sh,run}: use new local_export instead of local...export. this still requires calling echo_vars manually, because local_export is an alias (and must be, to define the var in the caller's context).
Aaron Marcuse-Kubitza
05:54 AM Revision 9236: lib/sh/util.sh: added local_export alias, which combines the effects of local and export
Aaron Marcuse-Kubitza
05:36 AM Revision 9235: lib/sh/util.sh: renamed params() to kw_params() to clarify the type of params it handles
Aaron Marcuse-Kubitza
05:33 AM Revision 9234: lib/sh/util.sh: logging: echo all keyword params by overriding params(). this requires removing manual echo_vars calls.
Aaron Marcuse-Kubitza
05:31 AM Revision 9233: lib/sh/util.sh: moved functions section right after exceptions, so that other sections can use it. in particular, func_override is needed by verbose output to override non-verbose versions of functions.
Aaron Marcuse-Kubitza
05:28 AM Revision 9232: lib/sh/util.sh: only try to echo vars that are set, to avoid `declare -p` "not found" errors
Aaron Marcuse-Kubitza
05:09 AM Revision 9231: *{.sh,run}: use new `params` in all functions that have keyword params, in order to remove their vars from the environment. note that functions that use $? (such as die()) must save it *before* running params, because params will overwrite $?.
Aaron Marcuse-Kubitza
04:28 AM Revision 9230: web/links/index.htm: updated to Firefox bookmarks. added bash and MySQL-related bookmarks, including MySQL doc subfolder.
Aaron Marcuse-Kubitza
04:11 AM Revision 9229: lib/sh/util.sh: params(): added doc comment
Aaron Marcuse-Kubitza
04:10 AM Revision 9228: lib/sh/util.sh: added params(), which removes keyword-param-only vars from the environment
Aaron Marcuse-Kubitza
03:50 AM Revision 9227: lib/sh/util.sh: added unexport() (note that `declare +x` won't work because it defines the var if it isn't set)
Aaron Marcuse-Kubitza
03:39 AM Revision 9226: lib/sh/util.sh: moved vars section to beginning so other sections can use it (it does not contain logging statements, so it doesn't need to be after verbose output)
Aaron Marcuse-Kubitza
03:34 AM Revision 9225: bugfix: lib/sh/util.sh: command(): limit cmd_log_fd directly instead of limiting log_fd (usually fd 2) and then redirecting cmd_log_fd to it. this ensures that stderr is not limited unless it's actually used as the cmd_log_fd, so that a non-logging stderr will still reach the user properly regardless of whether err_fd is set to something other than 2. (previously, without an err_fd at 22, closing fd 2 at the beginning of the set of redirections would leave no fd pointing to a place to echo errors to.)
Aaron Marcuse-Kubitza

05/11/2013

11:16 PM Revision 9224: lib/sh/util.sh: disable_logging(): use echo_eval instead of plain eval so the disabling of logging is nevertheless itself logged
Aaron Marcuse-Kubitza
11:11 PM Revision 9223: lib/sh/util.sh: command(): made usage message an end-of-line comment
Aaron Marcuse-Kubitza
11:10 PM Revision 9222: lib/sh/util.sh: inlined limit_log_fd into command() since it's only used once
Aaron Marcuse-Kubitza
10:22 PM Revision 9221: lib/sh/util.sh: log_err(), log_info(): use bolder symbols (*, #) to make the symbol stand out visually from the line of | above and below it
Aaron Marcuse-Kubitza
10:09 PM Revision 9220: lib/sh/util.sh: $verbosity: lowered the default from 3 to 2 because level 3 output (vars, etc.) is too verbose for the default output and is only needed for debugging
Aaron Marcuse-Kubitza
10:06 PM Revision 9219: lib/sh/util.sh: $log_level_indent: added space after | to visually separate the nesting levels and make the output look like an expandable tree (similar to a filesystem browser tree)
Aaron Marcuse-Kubitza
10:04 PM Revision 9218: lib/sh/util.sh: log_custom(): replace every non-whitespace character of $log_indent, and the first character of PS4, with the symbol, to make the symbol stand out on the line instead of the |||... before it
Aaron Marcuse-Kubitza
09:19 PM Revision 9217: lib/sh/util.sh: removed no longer used stdout2log_fd(). set $cmd_log_fd instead.
Aaron Marcuse-Kubitza
09:16 PM Revision 9216: lib/sh/util.sh: set $log_level_indent to | because this visually connects commands at the same indent by effectively connecting their + prefixes with a line of |
Aaron Marcuse-Kubitza
09:14 PM Revision 9215: lib/sh/archives.sh: zip(), unzip(): added missing echo_func
Aaron Marcuse-Kubitza
09:07 PM Revision 9214: lib/sh/util.sh: command(): use just one control var $cmd_log_fd instead of three flags (limit_log_fd, limit_stdout, stderr_is_errors) that indicated various common fd configurations. this is much clearer (you state which fd the common uses as its logging fd), more configurable (the logging fd can be any fd, not just 1 or 2), and more automatic (redirecting fd 2 to err_fd happens automatically if it isn't used for logging).
Aaron Marcuse-Kubitza
08:58 PM Revision 9213: bugfix: lib/sh/local.sh: psql(): need to use limit_stdout=1 instead of stderr_is_errors=1 because logging info is output to *stdout*, not *stderr*, in spite of the >&2 redirection, since the >&2 is applied *after* command()'s logging redirects. note that the >&2 can now be removed, because it will be performed by command().
Aaron Marcuse-Kubitza
08:16 PM Revision 9212: lib/sh/util.sh: added $log_fd and use it instead of 2 where the log_fd is intended
Aaron Marcuse-Kubitza
08:11 PM Revision 9211: lib/sh/util.sh: moved stdout2log_fd() to command verbose output subsection of verbose output since it is primarily intended for use with external commands
Aaron Marcuse-Kubitza
08:05 PM Revision 9210: lib/sh/util.sh: renamed limit_stderr to limit_log_fd, stdout2stderr to stdout2log_fd to clarify their purpose. log_fd is currently assumed to be 2.
Aaron Marcuse-Kubitza
08:00 PM Revision 9209: lib/sh/util.sh: added disable_logging() and use it instead of `exec 2>/dev/null`
Aaron Marcuse-Kubitza
07:54 PM Revision 9208: lib/sh/util.sh: added $err_fd global var and use it instead of the 22 magic value. it starts out at 2 until fd 22 has been set up, so commands that required 22 will now work before set_global_fds() is called.
Aaron Marcuse-Kubitza
07:11 PM Revision 9207: lib/sh/util.sh: shadow_fd(): don't require it to be run with log++ and instead let callers apply log++ if needed. in set_global_fds(), log++ can just be applied once for all the shadow_fd() calls.
Aaron Marcuse-Kubitza
07:09 PM Revision 9206: lib/sh/util.sh: shadow_fd(): instead of applying minor=1 (i.e. an additional log++) to echo_func and then log++ again separately to set_fd, just apply it to the entire function
Aaron Marcuse-Kubitza
07:07 PM Revision 9205: lib/sh/util.sh: set_fd(): don't increase the log_level for this function, and instead let callers increase it if needed
Aaron Marcuse-Kubitza
06:48 PM Revision 9204: lib/sh/util.sh: limit_stderr(): inlined log++ with the can_log call because it should only apply to it. this also makes it obvious that log++ should *not* apply outside of this function. this also enables the contents of this function to be used in an alias.
Aaron Marcuse-Kubitza
06:01 PM Revision 9203: lib/sh/util.sh: command(): use exec instead of `builtin command` to avoid spawning an extra process (since a subshell is already open)
Aaron Marcuse-Kubitza
05:43 PM Revision 9202: bugfix: lib/sh/util.sh: command(): only indent if command was logged
Aaron Marcuse-Kubitza
05:39 PM Revision 9201: bugfix: lib/sh/util.sh: .(): only indent if include statement was logged
Aaron Marcuse-Kubitza
05:29 PM Revision 9200: lib/sh/util.sh: limit_stderr(): moved $stdout2stderr to command() so that limit_stderr()'s only purpose is to log-limit stderr
Aaron Marcuse-Kubitza
05:23 PM Revision 9199: bugfix: lib/sh/util.sh: limit_stderr(): moved $stderr_is_errors default to command() where it's used
Aaron Marcuse-Kubitza
05:20 PM Revision 9198: lib/sh/util.sh: limit_stderr(): do stdout2stderr *after* limit_stderr to avoid having two similar `can_log ... >/dev/null` commands
Aaron Marcuse-Kubitza
05:09 PM Revision 9197: lib/sh/util.sh: echo_cmd(): removed no longer used removal of first builtin or command arg
Aaron Marcuse-Kubitza
05:07 PM Revision 9196: lib/sh/util.sh: inlined echo_run_prep into command() since it's now only used once
Aaron Marcuse-Kubitza
05:06 PM Revision 9195: lib/sh/util.sh: command(): removed no longer used $cmd_echoed flag
Aaron Marcuse-Kubitza
04:57 PM Revision 9194: *{.sh,run}: use new limit_stdout=1 instead of stdout2stderr=1 limit_stderr=1
Aaron Marcuse-Kubitza
04:56 PM Revision 9193: lib/sh/util.sh: command(): added limit_stdout=1 flag which implies stdout2stderr=1 limit_stderr=1
Aaron Marcuse-Kubitza
04:41 PM Revision 9192: lib/sh/util.sh: merged limit_stderr_cmd/limit_stdout_cmd into command(), using flag vars to control what limiting actions it needs to perform. in command invocations, this involves setting the appropriate flag vars instead of using a limit_std*_cmd alias.
Aaron Marcuse-Kubitza
04:37 PM Revision 9191: lib/sh/util.sh: added run_cmd
Aaron Marcuse-Kubitza
04:04 PM Revision 9190: lib/sh/util.sh: limit_stderr(): moved before command echoing because it does not depend on it and is a separate feature
Aaron Marcuse-Kubitza
03:47 PM Revision 9189: lib/sh/util.sh: run_args_cmd(): use $top_script instead of `"$(canon_rel_path "$0")"`
Aaron Marcuse-Kubitza
03:42 PM Revision 9188: lib/sh/util.sh: $top_script: run canon_rel_path on $0
Aaron Marcuse-Kubitza
03:35 PM Revision 9187: lib/sh/util.sh: use `minor=1 echo_func` instead of echo_minor_func because the syntax is simple enough that you don't need a separate alias for it
Aaron Marcuse-Kubitza
03:32 PM Revision 9186: lib/sh/util.sh: echo_func: support using minor=1 to increase the log_level that the function call is echoed at. use this in echo_minor_func.
Aaron Marcuse-Kubitza
03:28 PM Revision 9185: lib/sh/util.sh: echo_*func aliases: use simpler && syntax (with `|| true` to prevent errexit if the condition is false) instead of an if statement. this also allows prefixing the alias with var definitions.
Aaron Marcuse-Kubitza
03:27 PM Revision 9184: lib/sh/util.sh: echo_*func aliases: use simpler && syntax (with `|| true` to prevent errexit if the condition is false) instead of an if statement. this also allows prefixing the alias with var definitions.
Aaron Marcuse-Kubitza
03:18 PM Revision 9183: lib/sh/util.sh: echo_func: only indent if the function call was displayed. this avoids confusing extra whitespace when a function call causes an indent but the function call itself isn't printed.
Aaron Marcuse-Kubitza
03:10 PM Revision 9182: bugfix: lib/sh/util.sh: echo_minor_func: need to include "$@" in echo_func() call
Aaron Marcuse-Kubitza
03:08 PM Revision 9181: lib/sh/util.sh: echo_func(): added usage
Aaron Marcuse-Kubitza
03:07 PM Revision 9180: lib/sh/util.sh: echo_func: added usage
Aaron Marcuse-Kubitza
02:50 PM Revision 9179: lib/sh/util.sh: $log_level_indent: changed to a single space rather than two, to avoid very deep indents as the call tree becomes more deeply nested. due to the + at the beginning of every line, a single space is enough to visualize the structure of the call tree (unlike source code, which requires more whitespace because it doesn't have a call tree structure).
Aaron Marcuse-Kubitza
02:46 PM Revision 9178: lib/sh/util.sh: echo_*func aliases: increase the indent for the duration of the function call. this causes function calls to be displayed in an indented call-tree structure, which makes the verbose output much easier to understand.
Aaron Marcuse-Kubitza
02:43 PM Revision 9177: bugfix: lib/sh/util.sh: recursive aliases (i.e. aliases that call a command of the same name): use "" around the command name so it isn't reexpanded if the function using the alias is copied using copy_func(). (aliases will be reexpanded every time a function is redeclared since they appear unquoted in the function definition.) this is a bug in `declare -f` for which there is no fix, necessitating the "" alias workaround instead.
Aaron Marcuse-Kubitza
02:01 PM Revision 9176: lib/sh/util.sh: moved log-indent-related vars/aliases before they are first used (by log())
Aaron Marcuse-Kubitza
01:54 PM Revision 9175: lib/sh/util.sh: renamed inc_log_level/dec_log_level to shorter log++/log--
Aaron Marcuse-Kubitza
01:54 PM Revision 9174: lib/sh/util.sh: renamed inc_log_level/dec_log_level to shorter log++/log--
Aaron Marcuse-Kubitza
01:51 PM Revision 9173: lib/sh/util.sh: renamed cmd_indent to just indent now that indent is not being used for the PS4 level
Aaron Marcuse-Kubitza
01:47 PM Revision 9172: *{.sh,run}: multi-command aliases: enclose in {} so the alias can be used after ||
Aaron Marcuse-Kubitza
01:44 PM Revision 9171: lib/sh/util.sh: renamed indent->PS4++ , outdent->PS4-- because these relate just to the PS4 length, not to the (whitespace-based) log_indent
Aaron Marcuse-Kubitza
01:38 PM Revision 9170: lib/sh/util.sh: removed no longer used echo_run(). use command() instead.
Aaron Marcuse-Kubitza
01:36 PM Revision 9169: *{.sh,run}: use command instead of deprecated echo_run (don't prepend anything when the command is already aliased with `command`)
Aaron Marcuse-Kubitza
01:29 PM Revision 9168: lib/sh/util.sh: command(): perform echo_run_prep itself instead of requiring echo_run to be added in an alias. echo_run_prep will not be performed if limit_stderr_cmd() has performed it already outside the limit_stderr section. this requires using `builtin command` instead of "command" when you don't want the echoing (such as in include_guard_var()).
Aaron Marcuse-Kubitza
01:07 PM Revision 9167: *{.sh,run}: multi-command aliases: enclose in {} so the alias can be used after ||
Aaron Marcuse-Kubitza
12:56 PM Revision 9166: bugfix: lib/sh/util.sh: .(): also need to echo the . before the included file
Aaron Marcuse-Kubitza
12:48 PM Revision 9165: lib/sh/util.sh: added echo_params alias and use it instead of `log "$*"`
Aaron Marcuse-Kubitza
12:43 PM Revision 9164: lib/sh/util.sh: .(): echo the include at log_level 2 because it's an internal command
Aaron Marcuse-Kubitza
12:34 PM Revision 9163: lib/sh/util.sh: echo_cmd(): also remove `builtin` before the command name
Aaron Marcuse-Kubitza
12:32 PM Revision 9162: lib/sh/util.sh: echo_cmd(): don't remove -- after command, because it is added by the command() function instead and does not appear in the command sent to echo_run()
Aaron Marcuse-Kubitza
12:29 PM Revision 9161: lib/sh/util.sh: command alias: don't alias-expand next word, because the next word should only be interpreted as a command (part of the effect of the `command` builtin)
Aaron Marcuse-Kubitza
12:27 PM Revision 9160: lib/sh/util.sh: external commands: always use command rather than echo_run to indicate that a command is external, because command() ensures that what it invokes is not a shell function, and sets 2>&22 where needed
Aaron Marcuse-Kubitza
12:23 PM Revision 9159: lib/sh/util.sh: .(): use echo_run so that included files are echoed as they are included. note that echo_run uses cmd_indent, so the included files' load-time logging output will be indented according to the nesting level of the include.
Aaron Marcuse-Kubitza
12:20 PM Revision 9158: lib/sh/util.sh: moved .() command echoing > internal commands section
Aaron Marcuse-Kubitza
12:19 PM Revision 9157: lib/sh/util.sh: moved echo_run() to general command echoing section because it can also be used for internal commands
Aaron Marcuse-Kubitza
12:09 PM Revision 9156: removed no longer used inputs/GBIF/MySQL_export. use lib/sh/local.sh mysql_export_local() instead.
Aaron Marcuse-Kubitza
12:07 PM Revision 9155: inputs/GBIF/raw_occurrence_record/run: override table.tsv/make() instead of export_mysql()
Aaron Marcuse-Kubitza
12:06 PM Revision 9154: inputs/GBIF/table.run: table.tsv/make(): use new mysql_export_local instead of ./MySQL_export
Aaron Marcuse-Kubitza
12:05 PM Revision 9153: lib/sh/local.sh: added mysql_export_local()
Aaron Marcuse-Kubitza
12:05 PM Revision 9152: lib/sh/db.sh: added mysql_export(). documented that mysql_export_outfile() supports CSV, but requires the FILE privilege.
Aaron Marcuse-Kubitza
12:02 PM Revision 9151: lib/sh/db.sh: mysql(): use --column-names to ensure the output is formatted appropriately. note that --column-names is the default, but better to specify it to be sure.
Aaron Marcuse-Kubitza
12:01 PM Revision 9150: lib/sh/db.sh: mysql(): output_data mode: use --batch to ensure the output is formatted appropriately. note that --batch is the default when stdin is from a pipe, but better to specify it to be sure.
Aaron Marcuse-Kubitza
11:49 AM Revision 9149: *{.sh,run}: use new limit_stdout_cmd instead of `stdout2stderr=1 limit_stderr_cmd`
Aaron Marcuse-Kubitza
11:40 AM Revision 9148: lib/sh/util.sh: added limit_stdout_cmd alias
Aaron Marcuse-Kubitza
07:25 AM Revision 9147: lib/sh/make.sh: added $rm short var for $remake
Aaron Marcuse-Kubitza
07:22 AM Revision 9146: lib/sh/util.sh: verbosity: removed `declare -i verbosity` because this does not actually ensure that $verbosity is an integer; it only has "arithmetic evaluation [...] done when the variable is assigned to" (`help declare`)
Aaron Marcuse-Kubitza
07:20 AM Revision 9145: lib/sh/util.sh: verbosity: support setting this in alternate short-name var $vb
Aaron Marcuse-Kubitza
07:15 AM Revision 9144: lib/sh/util.sh: override the `.` builtin (a.k.a. `source`) and run canon_rel_path on the included script. this removes .. in the path when it's displayed in bash error messages.
Aaron Marcuse-Kubitza
06:58 AM Revision 9143: lib/sh/util.sh: log_err(): ensure errors are visible by using 2>&22. since this is uses log(), verbosity=1 also needs to be set.
Aaron Marcuse-Kubitza
06:50 AM Revision 9142: lib/sh/util.sh: $explicit_errors_only: explicitly set verbosity=0 to hide startup logging
Aaron Marcuse-Kubitza
06:46 AM Revision 9141: lib/sh/util.sh: limit_stderr(): if stdout goes to stderr (i.e. is logging info), assume that stderr_is_errors
Aaron Marcuse-Kubitza
06:41 AM Revision 9140: lib/sh/util.sh: $explicit_errors_only: documented that this option should only be used for testing the explicit error displaying, as otherwise, important output may be missed
Aaron Marcuse-Kubitza
06:39 AM Revision 9139: lib/sh/util.sh: $explicit_errors_only: added usage
Aaron Marcuse-Kubitza
06:38 AM Revision 9138: bugfix: lib/sh/util.sh: command(): updated $errors_only var name to $explicit_errors_only
Aaron Marcuse-Kubitza
06:27 AM Revision 9137: lib/sh/util.sh: $explicit_errors_only: noted that verbosity=0 displays everything that isn't explicitly hidden
Aaron Marcuse-Kubitza
06:24 AM Revision 9136: lib/sh/local.sh: psql(): use new stderr_is_errors=1 since psql's logging output comes on stdout, so stderr contains only errors. (the caveat of this is that the query output is normally mixed with logging info on stdout, and needs to be separated out to a different file descriptor using --output.)
Aaron Marcuse-Kubitza
06:18 AM Revision 9135: lib/sh/util.sh: added $explicit_errors_only env var which shows only explicitly-displayed errors (those which have been redirected to fd 22). note that most of the time, this has the same effect as `verbosity=0 script...`.
Aaron Marcuse-Kubitza
06:17 AM Revision 9134: lib/sh/util.sh: command(): support ensuring errors are visible by redirecting them to fd 22 (global stderr) when $stderr_is_errors is set
Aaron Marcuse-Kubitza
05:51 AM Revision 9133: lib/sh/util.sh: `command`: always prepend -- to the args list (to ensure that the command name is not interpreted as a `command` option), not just when the command alias is used
Aaron Marcuse-Kubitza
05:40 AM Revision 9132: lib/sh/util.sh: set_global_fds(): increase the log_level so the shadow_fd()/set_fd() function names aren't output at the default verbosity (3), and the redirection commands themselves aren't output at verbosity <= 2. set_global_fds() happens at the beginning of every script that uses util.sh, and is fairly predictable, so it isn't necessary to always echo these commands.
Aaron Marcuse-Kubitza
05:29 AM Revision 9131: lib/sh/local.sh: psql(): use new convention for command-specific alternate stdout
Aaron Marcuse-Kubitza
05:15 AM Revision 9130: lib/sh/make.sh: inline_make: use new convention for command-specific alternate stdin
Aaron Marcuse-Kubitza
05:13 AM Revision 9129: *{.sh,run}: removed extra space between function name and ()
Aaron Marcuse-Kubitza
05:12 AM Revision 9128: lib/sh/util.sh: streams: moved setting of global stdin/stdout/stderr into set_global_fds () function
Aaron Marcuse-Kubitza
05:05 AM Revision 9127: lib/sh/util.sh: streams: allow commands to access global stdin/stdout/stderr using fd 20/21/22. this works even when /dev/tty isn't available.
Aaron Marcuse-Kubitza
05:05 AM Revision 9126: lib/sh/util.sh: streams: added convention that fd 10/11/12 should be used for command-specific alternate stdin/stdout/stderr
Aaron Marcuse-Kubitza
05:04 AM Revision 9125: lib/sh/util.sh: added streams functions fd_exists, set_fd, shadow_fd, and helpers
Aaron Marcuse-Kubitza
05:01 AM Revision 9124: lib/sh/util.sh: added echo_minor_func alias
Aaron Marcuse-Kubitza
05:00 AM Revision 9123: lib/sh/util.sh: added echo_eval()
Aaron Marcuse-Kubitza
04:32 AM Revision 9122: lib/sh/util.sh: renamed require_exists() to require_not_exists() because the command actually checks if the file *doesn't* exist
Aaron Marcuse-Kubitza
02:26 AM Revision 9121: lib/sh/util.sh: echo_vars(): use log() instead of manually echoing the log indent, so that the log indent string ($log_indent$PS4) doesn't have to be maintained in several different places
Aaron Marcuse-Kubitza
02:23 AM Revision 9120: lib/sh/util.sh: logging: visually separate the log_level-based indenting from the external command indenting by using the standard `set -x` prefix (1st char of PS4) for log_level indenting and $log_level_indent (i.e. whitespace) for external command indenting
Aaron Marcuse-Kubitza
02:19 AM Revision 9119: lib/sh/util.sh: logging: separate the log_level-based indenting from the external command indenting so that the log_level-based indenting can use a different prefix. this involves propagating a new var, $log_indent, to invoked commands instead of $PS4, and always prepending $log_indent to $PS4 when printing log messages.
Aaron Marcuse-Kubitza
01:55 AM Revision 9118: lib/sh/util.sh: propagate $log_level_indent to invoked commands
Aaron Marcuse-Kubitza
01:54 AM Revision 9117: lib/sh/util.sh: renamed $log_indent to $log_level_indent to indicate that this is the string added at each level, rather than the entire indent
Aaron Marcuse-Kubitza
01:49 AM Revision 9116: bugfix: lib/sh/util.sh: limit_stderr(): switched back to running inc_log_level inside limit_stderr(), because it should only apply during the function for use by can_log. documented why inc_log_level needs to be inside limit_stderr().
Aaron Marcuse-Kubitza
01:44 AM Revision 9115: bugfix: lib/sh/util.sh: limit_stderr(): stdout2stderr mode: only limit stdout, since stderr may contain error messages (which should always be displayed)
Aaron Marcuse-Kubitza
01:32 AM Revision 9114: bugfix: lib/sh/util.sh: limit_stderr_cmd(): need to load new aliases before it so limit_stderr is expanded
Aaron Marcuse-Kubitza
01:31 AM Revision 9113: bugfix: lib/sh/util.sh: limit_stderr alias: need to inc_log_level before rather than after limit_stderr(), so that can_log uses the right verbosity (previously, inc_log_level was part of limit_stderr() itself, but was moved to the alias because its effects were needed outside the function)
Aaron Marcuse-Kubitza
01:25 AM Revision 9112: lib/sh/util.sh: stdout2stderr(): fixed doc comment after space-before-() search-and-replace
Aaron Marcuse-Kubitza
01:24 AM Revision 9111: lib/sh/util.sh: usage(): use `return` instead of `(exit ...)`
Aaron Marcuse-Kubitza
01:23 AM Revision 9110: lib/sh/util.sh: verbose output: put commands into subsections
Aaron Marcuse-Kubitza
01:10 AM Revision 9109: lib/sh/util.sh: echo_run(): factored prep code out into echo_run_prep alias, which can also be used in commands that mimic it, such as limit_stderr_cmd()
Aaron Marcuse-Kubitza
01:06 AM Revision 9108: bugfix: lib/sh/util.sh: limit_stderr(): moved indent to echo_run prep, because it only applies to commands invoked with echo_run
Aaron Marcuse-Kubitza
01:00 AM Revision 9107: bugfix: lib/sh/util.sh: limit_stderr(): need to apply inc_log_level in the caller so it won't get rolled back at the end of the function
Aaron Marcuse-Kubitza
12:51 AM Revision 9106: lib/sh/db.sh: mysql_export_outfile(): ensure newline between format info ($mysql_load_data_format) and rest of SELECT query
Aaron Marcuse-Kubitza
12:49 AM Revision 9105: lib/sh/db.sh: mysql_export_outfile(): ensure newline between SELECT columns list and INTO OUTFILE
Aaron Marcuse-Kubitza
12:44 AM Revision 9104: lib/sh/util.sh: limit_stderr_cmd(): moved stdout2stderr code to limit_stderr() because it is not specific to running limit_stderr on single commands
Aaron Marcuse-Kubitza
12:33 AM Revision 9103: lib/sh/util.sh: limit_stderr(): fixed doc comment after space-before-() search-and-replace
Aaron Marcuse-Kubitza
12:29 AM Revision 9102: lib/sh/db.sh: added mysql_export_outfile(), which uses SELECT ... INTO OUTFILE
Aaron Marcuse-Kubitza
12:22 AM Revision 9101: lib/sh/db.sh: mysql(): added $output_data option that turns off --verbose and uses echo_stdin instead so that queries are not echoed to stdout along with data. in --verbose mode, use limit_stderr_cmd with stdout2stderr=1 to redirect echoed queries through the logging mechanism.
Aaron Marcuse-Kubitza
12:17 AM Revision 9100: lib/sh/db.sh: mysql_cmd(): removed set_database because this is now performed by each caller before they use $database
Aaron Marcuse-Kubitza
12:15 AM Revision 9099: lib/sh/db.sh: mysql_cmd: get command name by adding $FUNCNAME in an alias instead of using ${FUNCNAME[1]} in the function, so that callers can also call mysql_cmd via a nested function, etc.
Aaron Marcuse-Kubitza
12:00 AM Revision 9098: lib/sh/util.sh: added ensure_nested_func()
Aaron Marcuse-Kubitza

05/10/2013

11:59 PM Revision 9097: lib/sh/util.sh: added contains()
Aaron Marcuse-Kubitza
11:57 PM Revision 9096: lib/sh/util.sh: command alias: also prepend -- to args to prevent a first arg starting with -- from being interpreted as an option to the `command` builtin
Aaron Marcuse-Kubitza
10:16 PM Revision 9095: lib/sh/db.sh: mysql_cmd(): moved setting of database to specific callers (mysql(), mysqldump()) because the syntax to set it differs between mysql* commands (e.g. mysql uses `--database=` while mysqldump uses `--databases ... --tables`)
Aaron Marcuse-Kubitza
10:13 PM Revision 9094: lib/sh/db.sh: added set_database alias, and use it in mysql_cmd()
Aaron Marcuse-Kubitza
10:11 PM Revision 9093: lib/sh/util.sh: import_vars alias and applicable aliases that use it: documented that when used inside another alias 2+ levels deep, it *must* be run inside a function. this is due to a strange bug in bash where $() expressions in 2-level aliases produce a syntax error when the alias is expanded outside a function.
Aaron Marcuse-Kubitza
10:07 PM Revision 9092: bugfix: lib/sh/util.sh: added back missing echo_func alias, which is needed to also include "$@" in the echoed function call
Aaron Marcuse-Kubitza
10:05 PM Revision 9091: lib/sh/util.sh: import_vars alias: use new get_prefix_vars()
Aaron Marcuse-Kubitza
10:04 PM Revision 9090: lib/sh/util.sh: add get_prefix_vars()
Aaron Marcuse-Kubitza
10:03 PM Revision 9089: lib/sh/util.sh: import_vars alias: use declare instead of local so it can be used outside a function
Aaron Marcuse-Kubitza
02:47 PM Revision 9088: lib/sh/db.sh: added mk_select alias and use it in pg_export()
Aaron Marcuse-Kubitza
02:36 PM Revision 9087: lib/sh/db.sh: pg_export(): always include the LIMIT value if specified, rather than only if the caller did not provide a full query
Aaron Marcuse-Kubitza
02:27 PM Revision 9086: lib/sh/db.sh: pg_export(): rtrim $query
Aaron Marcuse-Kubitza
02:22 PM Revision 9085: lib/sh/util.sh: added rtrim()
Aaron Marcuse-Kubitza
01:55 PM Revision 9084: lib/sh/db.sh: pg_export(): for simplicity, always use a SELECT statement as the source. take any source query in \$query without () instead of \$source with (). this will also help make pg_export uniform with MySQL.
Aaron Marcuse-Kubitza
01:34 PM Revision 9083: lib/sh/db.sh: renamed pg_copy_to() to pg_export() because it is also a general-purpose export command, which could have an analogous counterpart for MySQL
Aaron Marcuse-Kubitza
01:23 PM Revision 9082: bugfix: lib/sh/util.sh: moved self alias to beginning, before functions that use it
Aaron Marcuse-Kubitza
12:55 PM Revision 9081: lib/sh/util.sh: added repeat()
Aaron Marcuse-Kubitza
12:47 PM Revision 9080: lib/sh/util.sh: reverse(): use `i > 0` rather than `i >= 1` to match the `i < length` idiom used in forwards loops
Aaron Marcuse-Kubitza
12:43 PM Revision 9079: lib/sh/util.sh: moved strings section before verbose output so its commands can be used by the logging functions
Aaron Marcuse-Kubitza
12:32 PM Revision 9078: lib/sh/util.sh: echo_run() and derivatives (limit_stderr_cmd()): use new `indent` before the invoked command so its own logging messages, if any, are indented
Aaron Marcuse-Kubitza
12:27 PM Revision 9077: lib/sh/util.sh: propagate indent to invoked commands by exporting PS4
Aaron Marcuse-Kubitza
12:24 PM Revision 9076: lib/sh/util.sh: verbose output: added indent/outdent aliases and use them in inc_log_level/dec_log_level
Aaron Marcuse-Kubitza
12:14 PM Revision 9075: *{.sh,run}: replaced extern with command, a shell builtin that does the same thing. this is also part of dash (the Bourne shell on Linux).
Aaron Marcuse-Kubitza
12:08 PM Revision 9074: *{.sh,run}: removed extra space between function name and (), which is apparently not needed even though `help function` includes it. this greatly improves readability, including when function names are pasted into commit messages!
Aaron Marcuse-Kubitza
12:03 PM Revision 9073: lib/sh/util.sh: require_exists (): use die/log_info's new support for info messages using $type
Aaron Marcuse-Kubitza
12:00 PM Revision 9072: lib/sh/util.sh: die (): allow caller to specify a custom log message type (i.e. a suffix for log_* ())
Aaron Marcuse-Kubitza
11:58 AM Revision 9071: lib/sh/util.sh: log_e (): use new log_err
Aaron Marcuse-Kubitza
11:57 AM Revision 9070: lib/sh/util.sh: verbose output: added log_custom (), which modifies the symbol in PS4 to indicate the message type, and log_err ()/log_info (), which set specific symbols
Aaron Marcuse-Kubitza
11:55 AM Revision 9069: lib/sh/util.sh: moved die () into verbose output so it can use logging functions
Aaron Marcuse-Kubitza
11:53 AM Revision 9068: lib/sh/util.sh: moved functions after verbose output so copy_func () can use logging functions in its input checks
Aaron Marcuse-Kubitza
11:41 AM Revision 9067: lib/sh/util.sh: moved log_e () to verbose output section so it can use logging functions
Aaron Marcuse-Kubitza
11:29 AM Revision 9066: lib/sh/db.sh: mysqldump_diffable (): use pipe_delay to ensure that sed is printed after mysqldump
Aaron Marcuse-Kubitza
11:23 AM Revision 9065: lib/sh/util.sh: echo_cmd (): use new log ()
Aaron Marcuse-Kubitza
11:21 AM Revision 9064: lib/sh/util.sh: added log ()
Aaron Marcuse-Kubitza
10:53 AM Revision 9063: lib/sh/util.sh to_file, make.sh check_target_exists: use new require_exists to print message if file skipped because it already exists. this is useful for troubleshooting why certain make commands don't run.
Aaron Marcuse-Kubitza
10:51 AM Revision 9062: lib/sh/util.sh: added require_exists (), used to skip make commands for existing files
Aaron Marcuse-Kubitza
10:33 AM Revision 9061: *{.sh,run}: use `test !` instead of `! test` so that the ! is right next to the operator it's negating
Aaron Marcuse-Kubitza
10:31 AM Revision 9060: /run: geoscrub_input/make (): use new check_fake_target_exists to create the file only if it doesn't exist yet
Aaron Marcuse-Kubitza
10:31 AM Revision 9059: lib/sh/make.sh: added check_fake_target_exists (analogous to check_target_exists), which defers the target existence check until to_file (when the target name will presumably have been resolved to a path)
Aaron Marcuse-Kubitza
10:29 AM Revision 9058: lib/sh/util.sh: to_file (): support only running if the file does not exist by setting $if_not_exists
Aaron Marcuse-Kubitza
10:08 AM Revision 9057: lib/sh/util.sh: limit_stderr_cmd (): run cmd2rel_path explicitly here because echo_run is split apart rather than being run as echo_run ()
Aaron Marcuse-Kubitza
10:06 AM Revision 9056: lib/sh/util.sh: moved echo_cmd, echo_run before commands that use them
Aaron Marcuse-Kubitza
10:02 AM Revision 9055: lib/sh/make.sh: check_target_exists: if remaking, consider target not to exist
Aaron Marcuse-Kubitza
10:01 AM Revision 9054: lib/sh/make.sh: added remaking alias
Aaron Marcuse-Kubitza
09:45 AM Revision 9053: inputs/GBIF/table.run: table.tsv/make (): use echo_run instead of extern so that the command name is canonicalized properly
Aaron Marcuse-Kubitza
09:45 AM Revision 9052: lib/sh/util.sh: echo_run (): use new cmd2rel_path to resolve the command ($1)
Aaron Marcuse-Kubitza
09:44 AM Revision 9051: lib/sh/util.sh: added cmd2rel_path alias, which makes $1 a canon_rel_path if it's a filesystem path. this removes extra .. in the paths of invoked commands.
Aaron Marcuse-Kubitza
09:06 AM Revision 9050: *{.sh,run}: use shorter `test` instead of `test -n` and `test !` instead of `test -z` (http://www.gnu.org/software/bash/manual/bash.html#Bourne-Shell-Builtins > test)
Aaron Marcuse-Kubitza
08:52 AM Revision 9049: lib/sh/util.sh: echo_cmd (): don't include leading extern because it clutters up the output and is implied by the log_level
Aaron Marcuse-Kubitza
08:45 AM Revision 9048: lib/sh/util.sh: to_file (): run echo_func and `echo_vars stdout` to show what file the output is going to, since this information (i.e. redirects) isn't included in the logging output for the command itself
Aaron Marcuse-Kubitza
08:41 AM Revision 9047: lib/runscripts/table.run: remake_VegBIEN_mappings (): added public_schema_exists check, ported from lib/import.sh
Aaron Marcuse-Kubitza
08:40 AM Revision 9046: lib/sh/local.sh: added public_schema_exists (), ported from lib/import.sh
Aaron Marcuse-Kubitza
08:32 AM Revision 9045: *{.sh,run}: use new echo_stdout instead of echo_stdin where applicable, for clarity
Aaron Marcuse-Kubitza
08:31 AM Revision 9044: lib/sh/util.sh: added echo_stdout (currently just an alias of echo_stdin, because they are usable for the same purpose)
Aaron Marcuse-Kubitza
08:27 AM Revision 9043: /run: moved geoscrub_input export into separate geoscrub_input/make () target
Aaron Marcuse-Kubitza
08:24 AM Revision 9042: lib/sh/db.sh: pg_export_table_to_dir_no_header (): use to_file so that the file is autoremoved in case of error
Aaron Marcuse-Kubitza
07:51 AM Revision 9041: lib/sh/util.sh: let (): renamed to let! so that let can still be used to evaluate whether a numeric value is 0 (yes, an ! is allowed at the end of a command name)
Aaron Marcuse-Kubitza
07:44 AM Revision 9040: lib/sh/util.sh: moved canon_rel_path () into separate paths section
Aaron Marcuse-Kubitza
07:39 AM Revision 9039: bugfix: *{.sh,run}: use new func_override for runscript inheritance instead of invoking the overridden function as a (command-line) target of the parent runscript. this ensures that the overridden function is run in the *same* process as the calling function, so that $top_dir keeps the same value and runscript-relative paths continue to work.
Aaron Marcuse-Kubitza
07:30 AM Revision 9038: lib/sh/util.sh: added func_override (), for use in runscript inheritance
Aaron Marcuse-Kubitza
07:03 AM Revision 9037: lib/sh/util.sh: copy_func (): check that $from exists and $to does not exist (i.e. don't clobber existing functions). you can always first `unset -f` the function to get around the no-clobber restriction.
Aaron Marcuse-Kubitza
07:00 AM Revision 9036: lib/sh/util.sh: exceptions: added die ()
Aaron Marcuse-Kubitza
06:44 AM Revision 9035: lib/sh/util.sh: log_e (): restructured as an error handler to put after save_e rather than as a wrapper around the entire command. this allows it to be used with any kind of expression (such as boolean expressions with !), not just single commands.
Aaron Marcuse-Kubitza
06:35 AM Revision 9034: lib/sh/util.sh: log_e (): don't export $e to the calling context, since the caller can just use ` || { save_e; ...; }` if they need $e
Aaron Marcuse-Kubitza
06:31 AM Revision 9033: lib/sh/util.sh: calls to log_e: don't rely on log_e setting $e
Aaron Marcuse-Kubitza
06:30 AM Revision 9032: lib/sh/util.sh: added save_e, which now sets $e just locally
Aaron Marcuse-Kubitza
06:28 AM Revision 9031: lib/sh/util.sh: save_e: renamed to export_e because $e overwrites any previous value in the calling context
Aaron Marcuse-Kubitza
06:26 AM Revision 9030: lib/sh/util.sh: removed no longer used save_e_cmd
Aaron Marcuse-Kubitza
06:24 AM Revision 9029: lib/sh/util.sh: try (): use simpler save_e instead of save_e_cmd
Aaron Marcuse-Kubitza
06:19 AM Revision 9028: bugfix: lib/sh/util.sh: bool2int (): need to use try instead of save_e_cmd because save_e_cmd rethrows the error, which should instead just be stored in $e. this bug was not found in testing because bool2int was only used in $(), which errexit does not apply to.
Aaron Marcuse-Kubitza
06:15 AM Revision 9027: lib/sh/util.sh: exceptions: added save_e, now an alias for e=$?. added save_e/rethrow usage.
Aaron Marcuse-Kubitza
06:11 AM Revision 9026: lib/sh/util.sh: renamed save_e () to save_e_cmd () since it actually runs a command, in addition to saving $?
Aaron Marcuse-Kubitza
06:08 AM Revision 9025: lib/sh/util.sh: log_e (): rewrote to avoid using save_e, which will be repurposed
Aaron Marcuse-Kubitza
05:50 AM Revision 9024: lib/sh/util.sh: added func_exists ()
Aaron Marcuse-Kubitza
05:50 AM Revision 9023: lib/sh/util.sh: moved copy_func () after exceptions so it can use save_e/rethrow
Aaron Marcuse-Kubitza
05:44 AM Revision 9022: lib/sh/util.sh: added copy_func ()
Aaron Marcuse-Kubitza
05:39 AM Revision 9021: lib/sh/util.sh: save_e (): usage: added rethrow example
Aaron Marcuse-Kubitza
05:26 AM Revision 9020: bugfix: lib/sh/util.sh: bool2int (): need to load new aliases before it so that save_e will be expanded
Aaron Marcuse-Kubitza
05:24 AM Revision 9019: *{.sh,run}: put functions on one line where possible (and where they are not expected to expand)
Aaron Marcuse-Kubitza
05:19 AM Revision 9018: lib/sh/util.sh: added separate include guard around the include guard utils so they don't have to be redefined on every include of util.sh
Aaron Marcuse-Kubitza
05:06 AM Revision 9017: *{.sh,run}: added extra line before new sections to visually separate them. lib/sh/util.sh: added missing section headers.
Aaron Marcuse-Kubitza
05:01 AM Revision 9016: lib/sh/util.sh: split make utils out into separate make.sh
Aaron Marcuse-Kubitza
04:38 AM Revision 9015: lib/sh/util.sh: split archive (zip) utils out into separate archives.sh
Aaron Marcuse-Kubitza
04:24 AM Revision 9014: lib/sh/util.sh: split databases utils out into separate db.sh
Aaron Marcuse-Kubitza
04:05 AM Revision 9013: moved lib/*.sh to sh/ subdir so it's easier to find the .sh files among all the other lib/ files
Aaron Marcuse-Kubitza
03:32 AM Revision 9012: lib/util.sh: removed no longer used limit_stderr_extern. use `limit_stderr_cmd extern` instead.
Aaron Marcuse-Kubitza
03:29 AM Revision 9011: lib/util.sh: specify limit_stderr_cmd just for the (few) commands that need it, rather than for all commands, so that commands that use stderr to print important error messages don't have those error messages hidden when the verbosity is too low. (error messages should always be displayed, regardless of the verbosity.)
Aaron Marcuse-Kubitza
03:13 AM Revision 9010: lib/util.sh: limit_stderr_cmd (): only echo the command if it starts with echo_run. this requires adding echo_run before commands that use limit_stderr_cmd, such as limit_stderr_extern.
Aaron Marcuse-Kubitza
03:07 AM Revision 9009: lib/util.sh: limit_stderr_cmd (): remove echo_run from the command to run so that the command name isn't echoed twice
Aaron Marcuse-Kubitza
02:56 AM Revision 9008: lib/util.sh: limit_stderr_cmd: alias-expand command after it
Aaron Marcuse-Kubitza
02:53 AM Revision 9007: lib/util.sh: $verbosity: ensure it's an integer using `declare -i`
Aaron Marcuse-Kubitza
02:45 AM Revision 9006: lib/util.sh: grouped set verbosity statements together and commented them
Aaron Marcuse-Kubitza
02:43 AM Revision 9005: bugfix: lib/util.sh: verbose output: $verbosity defaults to $verbose (boolean) converted to integer. the previous set-default of $verbosity to $verbose has been removed because it came after `: "${verbosity=3}"` and thus didn't have an effect.
Aaron Marcuse-Kubitza
02:40 AM Revision 9004: lib/util.sh: added bool2int ()
Aaron Marcuse-Kubitza
02:32 AM Revision 9003: *{.sh,run}: use new isset
Aaron Marcuse-Kubitza
02:31 AM Revision 9002: lib/util.sh: added isset ()
Aaron Marcuse-Kubitza
02:28 AM Revision 9001: *{.sh,run}: use ${var+isset} instead of ${var+t} for clarity
Aaron Marcuse-Kubitza
02:22 AM Revision 9000: lib/util.sh: propagate the verbosity to invoked commands by exporting it
Aaron Marcuse-Kubitza
02:20 AM Revision 8999: lib/util.sh: run all commands verbosely by default, not just runscripts. this ensures verbose output for invoked commands like inputs/GBIF/MySQL_export.
Aaron Marcuse-Kubitza
02:15 AM Revision 8998: lib/util.sh: echo_stdin (): use new pipe_delay
Aaron Marcuse-Kubitza
02:12 AM Revision 8997: lib/util.sh: added pipe_delay (used as `cmd1 | { pipe_delay; cmd2; }`)
Aaron Marcuse-Kubitza
02:06 AM Revision 8996: lib/util.sh: limit_stderr_cmd (): echo the command like echo_run so that callers don't have to separately call echo_run. this reduces clutter of the nested aliases, ensures that the command is always echoed *outside* of the inner stderr-limiter (which has a different log_level), and avoids echoing "limit_stderr_cmd" itself as part of the command name.
Aaron Marcuse-Kubitza
01:47 AM Revision 8995: lib/util.sh: limit_stderr (): increase the log_level so that stderr of verbose commands can be turned off separately from the names of the commands themselves. it will now usually have log_level 2, indicating output that is useful primarily for debugging (this is the same as for shell function calls).
Aaron Marcuse-Kubitza
01:39 AM Revision 8994: lib/util.sh: renamed log_stderr* to limit_stderr* to reflect that stderr is limited (i.e. controlled) rather than logged
Aaron Marcuse-Kubitza
01:36 AM Revision 8993: lib/util.sh: echo_run_extern: renamed to log_stderr_extern since controlling stderr is its primary function
Aaron Marcuse-Kubitza
01:25 AM Revision 8992: inputs/GBIF/MySQL_export: only include WHERE clause if $filter is set. support configuring LIMIT/OFFSET.
Aaron Marcuse-Kubitza
01:00 AM Revision 8991: inputs/GBIF/table.run: table.tsv/make (): use new to_target to auto-delete $target on error
Aaron Marcuse-Kubitza
12:59 AM Revision 8990: inputs/GBIF/table.run: table.tsv/make (): use new check_target_exists
Aaron Marcuse-Kubitza
12:58 AM Revision 8989: lib/util.sh: make: added check_target_exists
Aaron Marcuse-Kubitza
12:42 AM Revision 8988: lib/util.sh: moved $top_file* to runscripts/util.run because they only apply to runscripts
Aaron Marcuse-Kubitza
12:36 AM Revision 8987: lib/util.sh: make: added to_target, which uses to_file on $target
Aaron Marcuse-Kubitza
12:35 AM Revision 8986: lib/util.sh: added to_file (), which auto-removes a command's output file on error (like make's .DELETE_ON_ERROR)
Aaron Marcuse-Kubitza
12:31 AM Revision 8985: lib/util.sh: echo_run_extern, extern: fixed/added comments to indicate that echo_run_extern echoes and controls stderr of *an* external command, while the extern alias does this for *all* external commands
Aaron Marcuse-Kubitza
12:28 AM Revision 8984: lib/util.sh: auto-echo common external commands, such as rm
Aaron Marcuse-Kubitza
12:24 AM Revision 8983: lib/util.sh: echo_run_extern, extern aliases: alias-expand next word (the command) by adding trailing space to alias def
Aaron Marcuse-Kubitza
12:21 AM Revision 8982: bugfix: lib/util.sh: exceptions: log_e: must include `declare e` in the alias and not when save_e is called, so that $e is a local var *of the caller*. this bug did not appear in testing because the save_e alias, which re-scopes $e within log_e (), was not expanded inside log_e () (since new aliases were not loaded between save_e and log_e ()).
Aaron Marcuse-Kubitza
12:12 AM Revision 8981: lib/util.sh: exceptions: added log_e (), which prints a "command exited with error" message (like make) when applicable
Aaron Marcuse-Kubitza
12:09 AM Revision 8980: lib/util.sh: exceptions: try_ (): renamed to try () and use the function keyword to distinguish it from the alias
Aaron Marcuse-Kubitza
12:06 AM Revision 8979: lib/util.sh: exceptions: try_ (): use new save_e
Aaron Marcuse-Kubitza
12:05 AM Revision 8978: lib/util.sh: exceptions: added save_e
Aaron Marcuse-Kubitza
12:01 AM Revision 8977: lib/util.sh: exceptions: end_try* aliases: use new rethrow*
Aaron Marcuse-Kubitza

05/09/2013

11:59 PM Revision 8976: lib/util.sh: exceptions: added rethrow, rethrow_subshell aliases
Aaron Marcuse-Kubitza
11:47 PM Revision 8975: lib/util.sh: always use '' rather than "" around alias definitions, to ensure that variables are evaluated at expand time rather than compile time
Aaron Marcuse-Kubitza
11:44 PM Revision 8974: lib/util.sh: zip_newer/unzip_newer: evaluate $no_force at alias expansion time rather than at alias compile time, so that the $no_force can be overridden in the context the alias is used in
Aaron Marcuse-Kubitza
11:18 PM Revision 8973: lib/util.sh: use `|| return` instead of `|| exit` because `|| exit` doesn't seem to work inside functions (it does not have the errexit effect). also, `|| return` has the advantage of *not* exiting the program if the caller used || after the command (i.e. as an error handler) to temporarily disable errexit.
Aaron Marcuse-Kubitza
11:11 PM Revision 8972: lib/runscripts/util.run: tell users to override run_args_cmd rather than this function to perform other commands, so that on_exit () can contain other exit-related processing that should *not* be overriden by the user
Aaron Marcuse-Kubitza
11:05 PM Revision 8971: lib/util.sh: removed "" around $?, $# because they are guaranteed to always be non-empty and contain no special chars
Aaron Marcuse-Kubitza
10:17 PM Revision 8970: lib/util.sh: say "last space" instead of "trailing space" so the comment will be more likely to fit at the end of the line
Aaron Marcuse-Kubitza
06:11 PM Revision 8969: lib/local.sh: removed mysql () since its functionality is now provided by mysql () in util.sh
Aaron Marcuse-Kubitza
05:59 PM Revision 8968: lib/util.sh: mysql_cmd (): $server: like $ssh_server, clear the value when it's equal to this machine's hostname
Aaron Marcuse-Kubitza
05:58 PM Revision 8967: lib/util.sh: mysql_cmd (): $ssh_server: use new localize_url to clear the value when it's equal to this machine's hostname
Aaron Marcuse-Kubitza
05:37 PM Revision 8966: lib/util.sh: added localize_url ()
Aaron Marcuse-Kubitza
05:36 PM Revision 8965: inputs/GBIF/table.run: table.tsv/make (): use extern so the MySQL_export command is echoed
Aaron Marcuse-Kubitza
05:24 PM Revision 8964: inputs/GBIF/table.run: table.tsv/make (): don't make the target if it already exists
Aaron Marcuse-Kubitza
05:16 PM Revision 8963: inputs/GBIF/table.run: table.tsv/make (): create the file directly rather than using inline_make, so that the command to create the file can use shell functions such as mysql
Aaron Marcuse-Kubitza
05:07 PM Revision 8962: bugfix: inputs/GBIF/table.run: table.tsv/make (): need to use $(target) (target's path *relative to curdir*) instead of just the filename, now that make does not change the directory
Aaron Marcuse-Kubitza
05:01 PM Revision 8961: inputs/GBIF/table.run: moved table.tsv creation to separate table.tsv/make () function
Aaron Marcuse-Kubitza
04:59 PM Revision 8960: lib/util.sh: echo_stdin (): display the echoed input after leading output of the echoed-to command, by using sleep with a small fractional # of seconds
Aaron Marcuse-Kubitza
04:38 PM Revision 8959: *{.sh,run}: put functions on one line where possible (and where they are not expected to expand)
Aaron Marcuse-Kubitza
04:32 PM Revision 8958: lib/local.sh: root_make (): use make () rather than extern make now that make () does not set its own current dir. using make () also automatically enables \$stdout2stderr, which is needed to redirect the echoed make commands.
Aaron Marcuse-Kubitza
04:26 PM Revision 8957: lib/util.sh: extern (): use exec to ensure that the executed command is in fact external, and not a shell builtin or other type of non-function command. this also simplifies the code because any shell function for the command does not need to be unset.
Aaron Marcuse-Kubitza
04:20 PM Revision 8956: lib/util.sh: make (): don't change the directory to $top_dir, because paths used by the caller will instead be relative to the current dir (note that in runscripts, paths are generally *not* absolute because of canon_rel_path ()). this requires manually specifying $top_dir when running make on a physical (rather than inlined) makefile.
Aaron Marcuse-Kubitza
04:01 PM Revision 8955: lib/util.sh: log_stderr_cmd (): replaced the `|| exit` after the () with `|| return`, for the same reason that this was a problem for extern () (see r8954)
Aaron Marcuse-Kubitza
03:55 PM Revision 8954: lib/util.sh: extern (): replaced the `|| exit` after the () with `|| return`, because `|| exit` strangely does not use $? as its exit status (even though `|| exit $?` works fine). in a shell function, it is actually sufficient, for errexit purposes, to use `|| return`, because the exit status is checked upon return from the function. it is in fact preferable to use return rather than exit, because then the caller can catch the exception using `|| true`.
Aaron Marcuse-Kubitza
03:51 PM Revision 8953: lib/util.sh: extern (): removed the `|| exit` inside the (), because this strangely prevents the exit status from being propagated to the exit status of the ()
Aaron Marcuse-Kubitza
03:44 PM Revision 8952: lib/util.sh: added explicit `set -o errexit` in case caller did not have -e in #! line
Aaron Marcuse-Kubitza
02:47 PM Revision 8951: inputs/GBIF/table.run: renamed MySQL_export () to export_mysql () to match schemas/VegCore/VegCore.ERD.mwb.run
Aaron Marcuse-Kubitza
02:21 PM Revision 8950: inputs/GBIF/table.run: renamed MySQL_export () to export_mysql () to match schemas/VegCore/VegCore.ERD.mwb.run
Aaron Marcuse-Kubitza
02:06 PM Revision 8949: inputs/FIA/_archive/2011-10-17/: set svn:ignore to *
Aaron Marcuse-Kubitza
01:52 PM Revision 8948: web/links/index.htm: updated to Firefox bookmarks. added Mac upgrade and recovery-related bookmarks.
Aaron Marcuse-Kubitza
12:07 PM Revision 8947: bugfix: lib/import.sh: remake_VegBIEN_mappings (): don't run if there is no public schema to perform the test on (e.g. if the staging tables have just been reinstalled)
Aaron Marcuse-Kubitza
12:06 PM Revision 8946: lib/import.sh: added public_schema_exists (), which just calls psql_script_vegbien (i.e. with the default search_path temp, public, etc.) and sees if returns an error
Aaron Marcuse-Kubitza
11:50 AM Revision 8945: /README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: when changes are made on vegbiendev: added step to download TWiki separately to your machine, to ensure that it gets backed up even if you don't sync the rest of the files. also, TWiki needs to be run with delete enabled to remove old session files, while the rest of the files should be run without delete.
Aaron Marcuse-Kubitza
08:28 AM Revision 8944: schemas/VegCore/VegCore.ERD.mwb.run: use my2pg's new sql_ascii mode
Aaron Marcuse-Kubitza
08:27 AM Revision 8943: bin/my2pg: support $sql_ascii mode, where varbinary is translated to text instead of bytea
Aaron Marcuse-Kubitza
08:27 AM Revision 8942: bin/my2pg: remove any () after text
Aaron Marcuse-Kubitza
08:23 AM Revision 8941: schemas/VegCore/VegCore.ERD.mwb.run: VegCore.my.sql/make (): leave out the DROP TABLE statements for now using --skip-add-drop-table
Aaron Marcuse-Kubitza
08:19 AM Revision 8940: added schemas/VegCore/VegCore.pg.sql.run with install () command (also part of all ())
Aaron Marcuse-Kubitza
08:19 AM Revision 8939: lib/local.sh: added root_make ()
Aaron Marcuse-Kubitza
08:15 AM Revision 8938: bin/my2pg: unpack custom types set('...')
Aaron Marcuse-Kubitza
08:12 AM Revision 8937: lib/local.sh: psql (): use fd 4 for the logging output instead of 3 because 3 is generally used for passing inline *input* commands (it is used this way by inline_make)
Aaron Marcuse-Kubitza
08:05 AM Revision 8936: schemas/VegCore/VegCore.ERD.mwb.run: sync (): don't pass "$@" to export_/import because they do not accept args, and the same set of args would not be meaningful for both
Aaron Marcuse-Kubitza
08:00 AM Revision 8935: schemas/VegCore/VegCore.ERD.mwb.run: use new $top_filename instead of $archive_filename
Aaron Marcuse-Kubitza
08:00 AM Revision 8934: lib/util.sh: added $top_script (alias for $0), $top_file, $top_filename. use new $top_file in $top_dir.
Aaron Marcuse-Kubitza
07:54 AM Revision 8933: schemas/VegCore/VegCore.ERD.mwb.run: put zip_newer/unzip_newer on the same line as the `cd "$top_dir"`
Aaron Marcuse-Kubitza
07:45 AM Revision 8932: bin/my2pg: translate varbinary to bytea
Aaron Marcuse-Kubitza
07:42 AM Revision 8931: schemas/VegCore/VegCore.ERD.mwb.run: added export_postgres (), which calls VegCore.pg.sql/make ()
Aaron Marcuse-Kubitza
07:29 AM Revision 8930: lib/util.sh: wrapper functions of external commands: use new self alias
Aaron Marcuse-Kubitza
07:28 AM Revision 8929: lib/util.sh: added self alias, for use with wrapper functions of external commands
Aaron Marcuse-Kubitza
07:18 AM Revision 8928: schemas/VegCore/VegCore.my.sql: regenerated using `VegCore.ERD.mwb.run export_mysql` (also part of the all target). note that this uses mysqldump, so the format and table order is different than it was for MySQL Workbench's custom export script.
Aaron Marcuse-Kubitza
07:05 AM Revision 8927: schemas/VegCore/VegCore.ERD.mwb.run: added export_mysql (), which runs VegCore.my.sql/make ()
Aaron Marcuse-Kubitza
07:05 AM Revision 8926: schemas/VegCore/VegCore.ERD.mwb.run: added export_mysql (), which runs VegCore.my.sql/make ()
Aaron Marcuse-Kubitza
07:03 AM Revision 8925: bugfix: lib/local.sh: $PATH: use $bin_dir_abs so the path will remain valid if the current directory is changed
Aaron Marcuse-Kubitza
07:02 AM Revision 8924: lib/local.sh: added \$bin_dir_abs
Aaron Marcuse-Kubitza
06:58 AM Revision 8923: bugfix: lib/util.sh: extern (): added `|| exit` after "$@" in addition to after the (), because this additional `|| exit` is apparently needed in order to propagate the exit status of the command to the exit status of the () and cause an error to abort the script properly
Aaron Marcuse-Kubitza
06:51 AM Revision 8922: lib/util.sh: echo_vars (): output vars with log_level 3 so they can be excluded separately from function calls to reduce clutter in the logging output. increase the default util.run verbosity so that vars are still printed by default.
Aaron Marcuse-Kubitza
06:47 AM Revision 8921: bugfix: lib/util.sh: echo_vars (): output each var separately so that each is prefixed by $PS4
Aaron Marcuse-Kubitza
06:39 AM Revision 8920: lib/util.sh: inc/dec_log_level: indent commands at higher log levels using $log_indent
Aaron Marcuse-Kubitza
06:39 AM Revision 8919: lib/util.sh: inc/dec_log_level: indent commands at higher log levels using $log_indent
Aaron Marcuse-Kubitza
06:31 AM Revision 8918: bugfix: lib/util.sh: make (): use stdout2stderr=1 so that commands make runs are properly output to stderr
Aaron Marcuse-Kubitza
06:21 AM Revision 8917: bugfix: lib/util.sh: include_guard_var (): use extern () when invoking sed so that this internal use of it isn't verbosely logged along with other external commands. note the "" around extern to refer to the function, not the alias.
Aaron Marcuse-Kubitza
06:16 AM Revision 8916: bugfix: lib/util.sh: zip/unzip (): use new $stdout2stderr instead of >&2 so that the redirect happens in the right order
Aaron Marcuse-Kubitza
06:15 AM Revision 8915: lib/util.sh: log_stderr_cmd (): support redirecting stdout to stderr *after*setting up the stderr redirect, so that stdout is also subject to that redirect
Aaron Marcuse-Kubitza
06:11 AM Revision 8914: lib/util.sh: added stdout2stderr ()
Aaron Marcuse-Kubitza
06:08 AM Revision 8913: lib/util.sh: changed echo_run_extern () to an alias
Aaron Marcuse-Kubitza
06:04 AM Revision 8912: lib/util.sh: echo_run_extern (): use new log_stderr_cmd
Aaron Marcuse-Kubitza
06:03 AM Revision 8911: lib/util.sh: added log_stderr_cmd ()
Aaron Marcuse-Kubitza
05:57 AM Revision 8910: lib/util.sh: use extern instead of env to run external commands, because it's simpler and auto-captures the logging output
Aaron Marcuse-Kubitza
05:44 AM Revision 8909: lib/util.sh: echo_func (): use the much simpler function keyword to avoid expanding the echo_func alias when defining the function, rather than going through the complex process of unaliasing and realiasing the echo_func alias
Aaron Marcuse-Kubitza
05:38 AM Revision 8908: lib/util.sh: mark env as always being an external command. it is no longer necessary to automatically echo it using echo_run because the extern alias does this now.
Aaron Marcuse-Kubitza
05:31 AM Revision 8907: lib/util.sh: automatically echo external commands. this requires using the function keyword when declaring extern () so that the new extern alias is not expanded in the function name.
Aaron Marcuse-Kubitza
05:28 AM Revision 8906: lib/util.sh: added echo_run_extern (), which automatically runs log_stderr to capture the external command's stderr
Aaron Marcuse-Kubitza
05:27 AM Revision 8905: lib/util.sh: added extern ()
Aaron Marcuse-Kubitza
05:05 AM Revision 8904: lib/util.sh: added log_stderr ()
Aaron Marcuse-Kubitza
04:54 AM Revision 8903: lib/util.sh: added end_try_subshell alias
Aaron Marcuse-Kubitza
04:29 AM Revision 8902: lib/util.sh: zip/unzip: redirect logging output to stderr
Aaron Marcuse-Kubitza
04:23 AM Revision 8901: lib/util.sh: use inc_log_level where logging info should only be output for verbosity >= 2: echo_func, echo_vars, echo_stdin
Aaron Marcuse-Kubitza
04:03 AM Revision 8900: lib/util.sh: verbose output: added inc_log_level, dec_log_level aliases which manipulate $verbosity. note that changes in $log_level are now instead indicated by moving $verbosity in the opposite direction.
Aaron Marcuse-Kubitza
04:01 AM Revision 8899: lib/util.sh: added let () wrapper that prevents it from triggering error-exit
Aaron Marcuse-Kubitza
03:16 AM Revision 8898: lib/util.sh: verbose output: use just one var ($verbosity) instead of two ($verbosity and $log_level), to track log level, for simplicity. $verbosity would now be subtracted from to alter the log level instead of $log_level being added to.
Aaron Marcuse-Kubitza
03:09 AM Revision 8897: lib/util.sh: use new can_log wherever logging info is output to stderr
Aaron Marcuse-Kubitza
03:05 AM Revision 8896: renamed lib/runscripts/local.run to lib/local.sh since the things it defines are not just for runscripts
Aaron Marcuse-Kubitza
02:53 AM Revision 8895: lib/util.sh: verbose output: added $log_level (must always be > 0 so verbosity=0 turns off all logging) and can_log (), which compares $log_level to $verbosity
Aaron Marcuse-Kubitza
02:32 AM Revision 8894: bugfix: lib/local.sh: updated $root_dir
Aaron Marcuse-Kubitza
02:31 AM Revision 8893: renamed lib/runscripts/local.run to lib/local.sh since the things it defines are not just for runscripts
Aaron Marcuse-Kubitza
02:22 AM Revision 8892: lib/runscripts/local.run: added mysqldump_local, which uses vegbiendev
Aaron Marcuse-Kubitza
02:21 AM Revision 8891: lib/runscripts/local.run: use postgres_compat mode because all our MySQL schemas should be usable as inputs to my2pg
Aaron Marcuse-Kubitza
02:20 AM Revision 8890: lib/runscripts/local.run: added connection vars for local (bien) user
Aaron Marcuse-Kubitza
02:18 AM Revision 8889: lib/util.sh: moved unalias inside include guard since it is not needed by anything *before* the include guard (even though it is used by commands *outside* the include guard, i.e. when the include guard is temporarily closed and reopened to load/unload aliases)
Aaron Marcuse-Kubitza
02:06 AM Revision 8888: lib/util.sh: use declare instead of local in aliases so that the aliases can also be used outside a function (declare will create a local var when used inside a function, and a global var otherwise)
Aaron Marcuse-Kubitza
02:00 AM Revision 8887: lib/util.sh: zip: ignore "zip has nothing to do" exit status as this is not an error
Aaron Marcuse-Kubitza
01:58 AM Revision 8886: lib/util.sh: added exception-handling (i.e. error-suppressing) functions
Aaron Marcuse-Kubitza
01:03 AM Revision 8885: bugfix: lib/util.sh: echo_func alias: moved it outside the include guard because the unalias statement is needs to be outside the include guard, and therefore so does the alias statement in order to always restore the alias
Aaron Marcuse-Kubitza
12:59 AM Revision 8884: lib/util.sh: renamed echo_func_ () back to echo_func () and instead add an unalias statement *before* the if statement containing the function definition. this is necessary because if statements are "compound commands", which have aliases expanded when their definition is read (i.e. at the beginning of the if statement, for the entire if statement), not when they are executed, and any aliases for function names in them need to be unaliased *before* the if statement containing the function definition. this is necessary even if the if statement is *not* executed, because the expansion of such aliases will cause syntax errors in the parsing of the if statement itself.
** note that this is preferable to the previous approach, because it causes bash to automatically fall back to the fu... Aaron Marcuse-Kubitza
12:45 AM Revision 8883: lib/util.sh: added unalias () override, which doesn't produce an error if the alias is undefined
Aaron Marcuse-Kubitza
12:04 AM Revision 8882: bugfix: lib/util.sh: renamed echo_func () to echo_func_ (), because you apparently can't have a function with the same name as an alias, even if the function is defined before the alias and an include guard prevents the function from being defined again after the alias is set. there may be some kind of alias-expanding preprocessing pass that bash only does in script mode, or maybe there is some other problem related to our use of aliases inside if statements.
Aaron Marcuse-Kubitza

05/08/2013

08:02 PM Revision 8881: *{.sh,run}: changed echo_func to an alias that includes the "$@", so that callers don't need to specify the "$@" manually. although the original echo_func function is still there, callers need to switch over to the alias at the same time as it's defined because otherwise the "$@" would be doubled, since echo_func refers to the alias instead.
Aaron Marcuse-Kubitza
07:21 PM Revision 8880: lib/util.sh: added set_make_vars alias, for use with make-target-style shell functions
Aaron Marcuse-Kubitza
06:37 PM Revision 8879: lib/util.sh: mysql_cmd (): interpret an empty string for $user, $database as if they were unset
Aaron Marcuse-Kubitza
06:34 PM Revision 8878: bugfix: lib/util.sh: mysql_cmd (): add --tables after specifying --databases so that future unnamed arguments are correctly interpreted as tables
Aaron Marcuse-Kubitza
06:33 PM Revision 8877: bugfix: lib/util.sh: mysql_cmd (): need to use `--databases ...` rather than `--database=...` to specify the database as a named argument
Aaron Marcuse-Kubitza
06:26 PM Revision 8876: bugfix: lib/util.sh: mysql_cmd (): need space before $database in command
Aaron Marcuse-Kubitza
06:23 PM Revision 8875: lib/util.sh: mysql_cmd (): use $schema as $database when provided
Aaron Marcuse-Kubitza
06:14 PM Revision 8874: lib/util.sh: mysql_cmd (): accept $database connection var
Aaron Marcuse-Kubitza
06:02 PM Revision 8873: *{.sh,run}: use just env instead of echo_run env now that env is an auto-echoing alias
Aaron Marcuse-Kubitza
05:56 PM Revision 8872: lib/util.sh: automatically echo commands that use env
Aaron Marcuse-Kubitza
05:55 PM Revision 8871: lib/util.sh: moved strings after verbose output so strings commands can use verbose output aliases
Aaron Marcuse-Kubitza
05:28 PM Revision 8870: lib/util.sh: mysql_cmd (): don't ssh if the ssh_server is this machine
Aaron Marcuse-Kubitza
05:27 PM Revision 8869: lib/util.sh: mysql_cmd (): allow specifying the ssh_dest as a separate ssh_server and ssh_user
Aaron Marcuse-Kubitza
05:08 PM Revision 8868: lib/util.sh: using prefixed connection vars: added use_local, use_local_remote aliases
Aaron Marcuse-Kubitza
04:42 PM Revision 8867: lib/util.sh: added mysqldump_diffable ()
Aaron Marcuse-Kubitza
04:41 PM Revision 8866: lib/util.sh: added mysqldump () wrapper, which does what my2pg_export does, but integrates with mysql_cmd and allows turning off postgres-compatible mode
Aaron Marcuse-Kubitza
04:15 PM Revision 8865: lib/util.sh: mysql_cmd (): use ${FUNCNAME[1]} to get the caller's command name instead of requiring the caller to pass the command name explicitly
Aaron Marcuse-Kubitza
03:56 PM Revision 8864: lib/util.sh: databases: added use_remote alias to use the remote_* connection vars (which uses new import_vars)
Aaron Marcuse-Kubitza
03:54 PM Revision 8863: lib/util.sh: added import_vars alias
Aaron Marcuse-Kubitza
03:53 PM Revision 8862: lib/util.sh: mysql_cmd (): added doc comment
Aaron Marcuse-Kubitza
03:52 PM Revision 8861: lib/util.sh: mysql (): use new mysql_cmd to auto-add connection/login opts when specified
Aaron Marcuse-Kubitza
03:50 PM Revision 8860: lib/util.sh: added mysql_cmd ()
Aaron Marcuse-Kubitza
03:48 PM Revision 8859: lib/util.sh: added local_inv alias
Aaron Marcuse-Kubitza
03:34 PM Revision 8858: lib/util.sh: zip/unzip_newer: support extracting/updating the file even if it has an older mtime by setting $force, using new set_inv to get a var to use to determine whether to include the -u flag
Aaron Marcuse-Kubitza
03:32 PM Revision 8857: lib/util.sh: added set_inv ()
Aaron Marcuse-Kubitza
03:30 PM Revision 8856: lib/util.sh: echo_vars (): prefix the declare output with "+ " because it's actually a command, not just an indication of the var's value
Aaron Marcuse-Kubitza
03:28 PM Revision 8855: lib/util.sh: added sed ()
Aaron Marcuse-Kubitza
03:23 PM Revision 8854: lib/util.sh: added section labels
Aaron Marcuse-Kubitza
10:57 AM Revision 8853: web/links/index.htm: removed broken favicons
Aaron Marcuse-Kubitza
10:55 AM Revision 8852: web/links/index.htm: removed broken favicons. this requires using the SQLite Manager Firefox add-on <https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/?src=search> to manually remove the favicon entries.
Aaron Marcuse-Kubitza
10:18 AM Revision 8851: lib/Firefox_bookmarks.reformat.csv: formatting fixes: don't put the src attribute of the favicon image on its own line, because it no longer contains long inline image data
Aaron Marcuse-Kubitza
10:15 AM Revision 8850: lib/Firefox_bookmarks.reformat.csv: favicon image: remove ICON attributes with inline image data, since these are no longer needed for the favicon images and this avoids cluttering up the svn diff
Aaron Marcuse-Kubitza
10:11 AM Revision 8849: lib/Firefox_bookmarks.reformat.csv: favicon image: use ICON_URI instead of ICON to fetch the favicon directly from the original site. this allows the favicons to be cached separately from the bookmarks page, so that the inline favicon contents don't need to be re-retrieved every time the bookmarks page is updated. this also avoids cluttering up the svn diff.
Aaron Marcuse-Kubitza
10:01 AM Revision 8848: web/links/index.htm: updated to Firefox bookmarks. added description of the formats supported by PostgreSQL's COPY command and their escape sequences.
Aaron Marcuse-Kubitza
08:44 AM Revision 8847: bugfix: schemas/vegbien.sql: geoscrub_input_view: only include places whose country is not NULL, because at least the country is required for geoscrubbing
Aaron Marcuse-Kubitza
08:34 AM Revision 8846: /README.TXT: Schema changes: Reinstall staging tables: added step to run `make schemas/public/install` after the datasources are installed. note that this must be run *after* the datasources are installed, because views in public depend on some of the datasources.
Aaron Marcuse-Kubitza

05/06/2013

07:10 AM Revision 8845: /README.TXT: Schema changes: Reinstall staging tables: use `make db` instead of mk_db, which includes the required schemas/install command. some datasources use the util schema in their postprocess scripts, so this must be installed before the staging tables.
Aaron Marcuse-Kubitza
07:05 AM Revision 8844: bugfix: /Makefile: moved schemas/install from install to db so that it is also run whenever the DB is reinstalled. schemas/install is needed by the staging tables reinstallation performed by reinstall_all.
Aaron Marcuse-Kubitza
06:33 AM Revision 8843: bugfix: /README.TXT: Schema changes: Reinstall staging tables: also need to run `make mk_db` before running reinstall_all
Aaron Marcuse-Kubitza
06:30 AM Revision 8842: /README.TXT: Schema changes: Reinstall staging tables: drop the saved previous DB at the end of the operation
Aaron Marcuse-Kubitza
06:18 AM Revision 8841: /README.TXT: Schema changes: Reinstall staging tables: run all of the local machine steps on vegbiendev, including the vegbien DB rename, to ensure that the existing data is saved in case the reinstall has errors
Aaron Marcuse-Kubitza
05:57 AM Revision 8840: /README.TXT: Schema changes: Reinstall staging tables: added step to first rename existing vegbien database, so it is not modified in the reinstallation
Aaron Marcuse-Kubitza
05:27 AM Revision 8839: inputs/FIA/: archived no longer used BIEN2 FIA data from the nimoy geoscrub DB
Aaron Marcuse-Kubitza
05:26 AM Revision 8838: inputs/FIA/_archive/: moved FIA_COND_unique, Organism into 2011-10-17/ subdir
Aaron Marcuse-Kubitza
04:44 AM Revision 8837: /README.TXT: Schema changes: Reinstall staging tables: added step to first run this on the local machine to check for any errors, before running it on the live DB, to prevent the live DB from becoming unrestorable in the case of bugs
Aaron Marcuse-Kubitza
04:27 AM Revision 8836: bugfix: bin/with_all: pipe yes to the command in case it makes any interactive prompts, as in e.g. the TNRS staging tables reinstall
Aaron Marcuse-Kubitza
04:25 AM Revision 8835: bugfix: inputs/input.Makefile: sql/install: manually specify $no_search_path option to psql_script_vegbien, which is added automatically in $(psqlNoSearchPath) but that uses psql_verbose_vegbien
Aaron Marcuse-Kubitza
04:12 AM Revision 8834: bugfix: *{.sh,run}: use `local` when setting default values for params to prevent these values from leaking into the calling context, where it may have unexpected effects (e.g. pg_copy_to would permanently set $source, which would then be used on the next invocation of pg_copy_to instead of regenerating $source from the next $table value). this involves using ${var-default} or ${var:-default} to return the value to the local statement instead of setting it directly with ${var=default} or ${var:=default}.
Aaron Marcuse-Kubitza
04:00 AM Revision 8833: *{.sh,run}: commands that take named arguments: usage messages: changed to use simpler bash syntax for specifying command-specific env vars, which does not require a subshell ("The environment for any simple command or function may be augmented temporarily by prefixing it with parameter assignments" <http://www.gnu.org/software/bash/manual/bash.html#Environment>)
Aaron Marcuse-Kubitza
03:14 AM Revision 8832: schemas/pg_hba*.conf: removed trailing whitespace
Aaron Marcuse-Kubitza
03:10 AM Revision 8831: schemas/pg_hba*.conf: allow the bien group to access all databases, including test (on the Mac). note that this does not grant access to the vegbank DB on vegbiendev, because that is owned by postgres.
Aaron Marcuse-Kubitza
02:32 AM Revision 8830: web/links/index.htm: explicitly separate page's description from author comments where needed
Aaron Marcuse-Kubitza
02:29 AM Revision 8829: web/links/index.htm: use - instead of --- to separate the page's description from the author comments because it's faster to type and one hyphen is sufficient
Aaron Marcuse-Kubitza
02:21 AM Revision 8828: lib/Firefox_bookmarks.reformat.csv: labeling page's description: don't match descriptions with a leading blank line as an author comment, because sometimes page descriptions themselves start with a blank line. instead, author comments at the beginning of bookmark descriptions should always start with a " or be preceded by a line containing only hyphens (-). note that it was never reliable to use a blank line for this purpose, because it would be trimmed out upon reimport of the HTML file into Firefox.
Aaron Marcuse-Kubitza
02:17 AM Revision 8827: web/links/index.htm: updated to Firefox bookmarks
Aaron Marcuse-Kubitza

05/05/2013

10:35 PM Revision 8826: inputs/NVS/*/map.csv: Taxon Growth Form: use new _lowercase() to avoid needing to manually map each value that is already valid but just needs to be lowercased
Aaron Marcuse-Kubitza
10:28 PM Revision 8825: schemas/util.sql: added _lowercase()
Aaron Marcuse-Kubitza
09:05 PM Revision 8824: added inputs/CTFS/ERD/ VegPath location
Aaron Marcuse-Kubitza
11:57 AM Revision 8823: added inputs/MO/_src/_README.TXT with warning that the refresh provided to us is missing several columns from the original extract: CollectionCode, CatalogNumber, IndividualCount
Aaron Marcuse-Kubitza
10:59 AM Revision 8822: schemas/VegCore/VegCore.ERD.mwb: reduced MaxColumnsDisplayed to 10 to prevent wide tables from taking up too much space in the ERD. this was a big problem in the VegBIEN ERD, which we are hoping to prevent in the VegCore ERD as first-class columns get added to the tables. note that it's very important to ensure that the first-class columns are sorted *in importance order*, so that the 10 most important are visible in the ERD.
Aaron Marcuse-Kubitza
10:34 AM Revision 8821: schemas/VegCore/VegCore.ERD.mwb: renamed specimen_holder_institution to specimenholder_institution because specimenholder is one word
Aaron Marcuse-Kubitza
10:12 AM Revision 8820: lib/sql_io.py: put_table(): Calling wrapper function: adding pkey or index on the resulting table: don't display warning if a pkey can't be added, because this is actually a legitimate situation when the called function is set-returning and can return multiple rows for one input. having this as a warning results in spurious warnings in the automated tests (which look confusingly like ignored errors because Python warnings include debugging context information). e.g. `make inputs/Madidi/IndividualObservation/test.by_col.xml` causes this error in the sourcelist->sourcename splitting step (which of course can produce multiple specimenholder institutions).
*** note that a *diff* will continue to be produced (but now without an error message), because row-based import does... Aaron Marcuse-Kubitza
10:00 AM Revision 8819: bugfix: lib/sql.py: parse_exception(): typed_name_re (used by MissingCastException, etc.): don't require "" around the value, because function names in "function does not exist" exceptions are (now?) output without them. this exception did *not* occur in the last import, but *does* occur in the automated testing and when running the associated query in pgAdmin (in both Linux and Mac OS X).
Aaron Marcuse-Kubitza
04:30 AM Revision 8818: bugfix: /Makefile: apache-Linux: use the standard apt-get syntax for Makefiles (`-sudo apt-get --yes install`) instead of just `apt-get install`
Aaron Marcuse-Kubitza
04:24 AM Revision 8817: /Makefile: added Apache 2.4 installation
Aaron Marcuse-Kubitza
03:57 AM Revision 8816: lib/util.sh: added join ()
Aaron Marcuse-Kubitza
02:58 AM Revision 8815: web/.twiki/data/Main/TWikiPreferences.txt: updated with current settings
Aaron Marcuse-Kubitza
02:55 AM Revision 8814: put web/.twiki/data/Main/TWikiPreferences.txt under version control. note that this file (*Main*/TWikiPreferences.txt) is the correct file to change the settings in, not *TWiki*/TWikiPreferences.txt.
Aaron Marcuse-Kubitza
02:33 AM Revision 8813: put web/.twiki/data/TWiki/TWikiPreferences.txt under version control
Aaron Marcuse-Kubitza
01:58 AM Revision 8812: inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt: twikibuttons_formats: added blockquote format
Aaron Marcuse-Kubitza
01:47 AM Revision 8811: inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt: set inline_styles to false so that markup formatting (underline, strikethrough) gets formatted using HTML tags (<strike>, etc.) instead of CSS styles ("text-decoration: line-through", etc.)
Aaron Marcuse-Kubitza
01:26 AM Revision 8810: inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt: theme_advanced_buttons1: added underline, strikethrough buttons
Aaron Marcuse-Kubitza
01:25 AM Revision 8809: put inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt under version control
Aaron Marcuse-Kubitza

05/02/2013

11:07 PM Revision 8808: /run: export_ (): pass "$@" to invoked functions since it may contain psql connection or config info
Aaron Marcuse-Kubitza
11:06 PM Revision 8807: lib/util.sh: pg_* (): pass "$@" to invoked functions since it may contain psql connection or config info
Aaron Marcuse-Kubitza
10:59 PM Revision 8806: bugfix: lib/util.sh: pg_header (): use limit=0 to prevent COPY TO from returning all rows in the table (not an issue for empty tables!)
Aaron Marcuse-Kubitza
10:53 PM Revision 8805: lib/util.sh: pg_copy_to (): support sources besides whole tables (such as custom queries and tables with columns). support limiting the # of rows retrieved when using whole tables.
Aaron Marcuse-Kubitza
10:01 PM Revision 8804: lib/runscripts/import.run: added export_ () command, which is run after import in all ()
Aaron Marcuse-Kubitza
09:57 PM Revision 8803: bin/psql_verbose_vegbien: also echo queries issued by backslash commands (--echo-hidden)
Aaron Marcuse-Kubitza
09:34 PM Revision 8802: /run: make config vars such as $schema overridable by the environment, to allow pointing the export script at a different public schema version
Aaron Marcuse-Kubitza
08:53 PM Revision 8801: inputs/input.Makefile: SVN: add, %/add: */logs: also svn:ignore *.gz, used for compressed log files
Aaron Marcuse-Kubitza
08:50 PM Revision 8800: added inputs/HIBG/Specimen/header.csv
Aaron Marcuse-Kubitza
08:50 PM Revision 8799: added inputs/HVAA/Specimen/Herbario_occur_1360871068.csv.md5
Aaron Marcuse-Kubitza
08:40 PM Revision 8798: exports/: svn:ignore *.csv
Aaron Marcuse-Kubitza
08:39 PM Revision 8797: added /run, whose export_ command exports DB tables (currently the geoscrub_input table) to CSV. column information is included in the filename when the export format is specified not to contain a header row.
Aaron Marcuse-Kubitza
08:35 PM Revision 8796: lib/runscripts/local.run: moved log_sql (), pg_* () to lib/util.sh because these do not depend on local external scripts in bin/
Aaron Marcuse-Kubitza
08:27 PM Revision 8795: lib/runscripts/local.run: added pg_copy_to (), pg_header (), pg_export_table_no_header (), pg_export_table_to_dir_no_header ()
Aaron Marcuse-Kubitza
08:26 PM Revision 8794: lib/runscripts/local.run: added mk_schema_esc, mk_table_esc aliases as shortcuts for `local *_esc; mk_esc_name *`
Aaron Marcuse-Kubitza
08:09 PM Revision 8793: lib/util.sh: self_not_included (): added support for loading newly-defined aliases for use in functions in the same file, and documentation for how to do this
Aaron Marcuse-Kubitza
02:49 PM Revision 8792: lib/runscripts/local.run: psql (): output only the query results to stdout (using --output), and redirect everything else to stderr
Aaron Marcuse-Kubitza
02:26 PM Revision 8791: added /exports/
Aaron Marcuse-Kubitza
02:24 PM Revision 8790: lib/runscripts/local.run: added log_sql (). mysql (): don't echo SQL commands when not in verbose mode. this is needed to support commands that use the query result in a $() expression, and should not have echoed commands cluttering up stdout. (unfortunately, the SQL commands are echoed to stdout rather than stderr.)
Aaron Marcuse-Kubitza
01:52 PM Revision 8789: schemas/vegbien.sql: range_modeling_input: use analytical_stem instead of analytical_stem_view because it takes a long time (~2 min) to get a row from the left-joined view, due to the large number of joins and large number of rows in the joined tables
Aaron Marcuse-Kubitza
01:41 PM Revision 8788: web/links/index.htm: updated to Firefox bookmarks. grouped related items together.
Aaron Marcuse-Kubitza
01:28 PM Revision 8787: web/links/index.htm: author comments at beginning of bookmark desc: start with a line containing only dashes (-) rather than a blank line, because a newline will be trimmed away on reimport into Firefox, such that reexporting it will lose the author comment indicator
Aaron Marcuse-Kubitza
01:24 PM Revision 8786: lib/Firefox_bookmarks.reformat.csv: label page's description: support starting a description with a line containing only dashes (-) to indicate author comments. this is better than a newline, because a newline will be trimmed away on reimport into Firefox, such that reexporting it will lose the author comment indicator
Aaron Marcuse-Kubitza
01:16 PM Revision 8785: lib/Firefox_bookmarks.reformat.csv: add hyperlinked HTML anchors for folders: switch back to using <a href=...> instead of onclick, but put the <a> tag *around* the <H3> rather than inside it so that Firefox can correctly parse out the folder name when importing the bookmarks
Aaron Marcuse-Kubitza
12:52 PM Revision 8784: lib/Firefox_bookmarks.reformat.csv: add hyperlinked HTML anchors for folders: use `onclick="javascript:document.location = '#...'"` instead of <a href=...> to allow clicking the header to go to the anchor, so that the <a> tag doesn't prevent Firefox from parsing the folder name
Aaron Marcuse-Kubitza
12:42 PM Revision 8783: bugfix: lib/Firefox_bookmarks.reformat.csv: add hyperlinked HTML anchors for folders: moved <a name=.../> outside the <H3> so that Firefox can correctly parse out the folder name when importing the bookmarks
Aaron Marcuse-Kubitza
12:27 PM Revision 8782: web/links/index.htm: updated to Firefox bookmarks. grouped related items together.
Aaron Marcuse-Kubitza
12:19 PM Revision 8781: lib/Firefox_bookmarks.reformat.csv: indexing: add hyperlinked HTML anchors for folders
Aaron Marcuse-Kubitza
11:50 AM Revision 8780: web/links/index.htm: updated to Firefox bookmarks
Aaron Marcuse-Kubitza
08:30 AM Revision 8779: schemas/vegbien.sql: added range_modeling_input view
Aaron Marcuse-Kubitza
08:21 AM Revision 8778: schemas/vegbien.sql: analytical_stem_view: order by datasource name so that the rows have a predictable rather than random ordering
Aaron Marcuse-Kubitza
06:14 AM Revision 8777: bugfix: lib/util.sh: moved mysql (), psql () to local.run because they use local commands that are only available after local.run sets the PATH. retain a mysql () function in util.sh that adds the --verbose option to mysql.
Aaron Marcuse-Kubitza
05:44 AM Revision 8776: lib/util.sh: removed "$bin_dir"/ before commands because the bin dir is now in the PATH
Aaron Marcuse-Kubitza
05:13 AM Revision 8775: lib/runscripts/table.run: moved general DB commands to lib/util.sh
Aaron Marcuse-Kubitza
04:47 AM Revision 8774: moved lib/*.run into runscripts/ subdir so runscripts are grouped together and easier to find rather than being scattered throughout lib/
Aaron Marcuse-Kubitza
04:19 AM Revision 8773: bugfix: lib/util.sh: canon_rel_path (): use $(pwd -P) instead of $PWD because symlinks are resolved by realpath (readlink -f), so they also need to be resolved in the current dir, but $PWD does not contain the symlinks-resolved version of the current dir
Aaron Marcuse-Kubitza
03:48 AM Revision 8772: inputs/GBIF/MySQL_export: use lib/util.sh instead of util.run now that the non-runscript-specific functions have been separated out into it. this ensures that MySQL_export is not unintentionally treated as a runscript (which functions differently from a normal shell script).
Aaron Marcuse-Kubitza
02:46 AM Revision 8771: /.htaccess: use canonical URL without symlinks
Aaron Marcuse-Kubitza
02:44 AM Revision 8770: added /.htaccess to create the right self-referential URL since / is outside the document root (linked from /vegbiendev/fs/)
Aaron Marcuse-Kubitza
02:14 AM Revision 8769: schemas/VegCore/VegCore.ERD.mwb: TNRS taxonomic scrubbing steps: moved labels to the *start* of the arrow they label, so that the user can more easily follow the arrows from step to step by looking for an outward-pointing arrow with a label
Aaron Marcuse-Kubitza
01:48 AM Revision 8768: added inputs/GBIF/_MySQL/MySQL.data.sql.gz.md5
Aaron Marcuse-Kubitza
12:46 AM Revision 8767: bugfix: lib/common.Makefile: Compression: %.gz <-> %: only run command if target does not exist, to avoid overwriting the target when the compressed or uncompressed version is newer than it. the difference in mtimes can arise when one file is created after the other, and should not cause the opposite operation to be performed to try to make the other file up-to-date (leading to an infinite back-and-forth of creating one file from the other).
Aaron Marcuse-Kubitza

04/30/2013

11:13 PM Revision 8766: web/links/index.htm: updated to Firefox bookmarks. the VegPath link now includes the new favicon.
Aaron Marcuse-Kubitza
11:12 PM Revision 8765: added web/favicon.ico generated from schemas/BIEN_logo.png
Aaron Marcuse-Kubitza
11:01 PM Revision 8764: web/main.conf: added SSL-protected site for use with password-protected areas of the site, so that the user's password is not transmitted in plaintext. (because the SSL certificate is self-signed, it will unfortunately display the Confirm Security Exception message in Firefox.) note that digest authentication cannot be used to avoid plaintext passwords, because it requires knowing the user's original password to generate the digest, but we have only the MD5 sums in /etc/shadow.
Aaron Marcuse-Kubitza
09:26 PM Revision 8763: web/links/index.htm: updated to Firefox bookmarks
Aaron Marcuse-Kubitza
08:08 PM Revision 8762: web/links/index.htm: put all bookmarks into one subfolder named BIEN links, to facilitate importing them all into one folder on another machine. this also helps when upgrading to a newer version of the bookmarks, because the previous version must be deleted and this now requires deleting only a single folder.
Aaron Marcuse-Kubitza
08:02 PM Revision 8761: lib/Firefox_bookmarks.reformat.csv: use <H3 style="font-size: 200%;"> instead of <H1> to make the first folder's name the page title, so that Firefox can reimport the first folder as a bookmarks folder
Aaron Marcuse-Kubitza
07:42 PM Revision 8760: bugfix: lib/Firefox_bookmarks.reformat.csv: make first folder name the page title: added back replacement string
Aaron Marcuse-Kubitza
07:40 PM Revision 8759: web/links/index.htm.run: clean_up (): removed command to add page title because this is now done by lib/Firefox_bookmarks.reformat.csv
Aaron Marcuse-Kubitza
07:40 PM Revision 8758: lib/Firefox_bookmarks.reformat.csv: use page title as window title
Aaron Marcuse-Kubitza
07:31 PM Revision 8757: lib/Firefox_bookmarks.reformat.csv: remove previous page title
Aaron Marcuse-Kubitza
07:29 PM Revision 8756: lib/Firefox_bookmarks.reformat.csv: make first folder name the page title
Aaron Marcuse-Kubitza
07:21 PM Revision 8755: lib/Firefox_bookmarks.reformat.csv: move Bookmarks Toolbar inside first folder instead of before it. this allows storing all bookmarks in one subfolder so that when the web page is imported into another computer's bookmarks, the imported bookmarks are all together in one folder.
Aaron Marcuse-Kubitza
07:16 PM Revision 8754: bugfix: lib/Firefox_bookmarks.reformat.csv: refactored regexps to account for Python applying (?s) to the entire regexp, even if it appears inside a () subgroup. the Python behavior is in contrast to the standard regexp behavior used by Java, etc. where (?s) applies only to the subgroup it appears in.
Aaron Marcuse-Kubitza
06:19 PM Revision 8753: web/links/index.htm: updated to Firefox bookmarks. resources: put unsorted links into subfolders.
Aaron Marcuse-Kubitza
06:11 PM Revision 8752: bugfix: lib/Firefox_bookmarks.reformat.csv: add <style>: format bookmark description as plaintext: use `white-space: pre-wrap` instead of pre in order to wrap long lines
Aaron Marcuse-Kubitza
06:07 PM Revision 8751: bugfix: lib/Firefox_bookmarks.reformat.csv: favicon images: made replacements idempotent so that the favicons would not be deleted the next time the web/links/index.htm.run is run (due to a newline being added before ICON= )
Aaron Marcuse-Kubitza
05:52 PM Revision 8750: lib/Firefox_bookmarks.reformat.csv: favicon images: display placeholder favicon image for every link to ensure uniform alignment
Aaron Marcuse-Kubitza
05:45 PM Revision 8749: added web/links/blank.gif symlink to web/blank.gif for shorter blank.gif URLs
Aaron Marcuse-Kubitza
05:43 PM Revision 8748: added web/blank.gif
Aaron Marcuse-Kubitza
05:14 PM Revision 8747: lib/Firefox_bookmarks.reformat.csv: display ICON attributes as separate images. note that this uses the "data:" embedded image contents provided in the bookmarks file, so the images load instantly along with the page instead of needing to be separately retrieved from each server.
Aaron Marcuse-Kubitza
04:45 PM Revision 8746: lib/Firefox_bookmarks.reformat.csv: instead of removing ICON attributes with favicon contents, put them on their own line so the rest of the line is easier to see
Aaron Marcuse-Kubitza
04:35 PM Revision 8745: lib/Firefox_bookmarks.reformat.csv: divided replacements into sections
Aaron Marcuse-Kubitza
04:27 PM Revision 8744: lib/Firefox_bookmarks.reformat.csv: move Bookmarks Toolbar to beginning, remove its desc, and rename it to Quick links
Aaron Marcuse-Kubitza
04:13 PM Revision 8743: web/links/index.htm: removed extra divider after Recent Tags, Recently Bookmarked because section dividers interfere with regexp matching and the Bookmarks Toolbar is already visually separate from them
Aaron Marcuse-Kubitza
04:09 PM Revision 8742: web/links/index.htm: moved Recent Tags, Recently Bookmarked special folders to end because they are Firefox-specific and not part of the BIEN links (they also do not work in a browser because they are Firefox-specific query bookmarks instead of actual folders)
Aaron Marcuse-Kubitza
03:32 PM Revision 8741: web/links/index.htm.run: clean_up (): rename the page title from "Bookmarks" to "BIEN links"
Aaron Marcuse-Kubitza
03:18 PM Revision 8740: bin/repl: skip header instead of parsing it, since its columns are not used. this also allows using just a blank line as the header when the column names are not important.
Aaron Marcuse-Kubitza
03:05 PM Revision 8739: web/links/index.htm: use initial blank line instead of previous "-" workaround to indicate author comment at start of bookmark desc
Aaron Marcuse-Kubitza
03:04 PM Revision 8738: web/links/index.htm: use initial blank line to indicate author comment at start of bookmark desc
Aaron Marcuse-Kubitza
02:59 PM Revision 8737: lib/Firefox_bookmarks.reformat.csv: label page's description: don't label a bookmark desc starting with a blank line (indicating an author-added comment) as a page's description
Aaron Marcuse-Kubitza
02:51 PM Revision 8736: lib/Firefox_bookmarks.reformat.csv: label page's description: don't label an author-added quotation (which starts with " ) as the page's description
Aaron Marcuse-Kubitza
02:33 PM Revision 8735: bin/in_place: set the permissions of the new file to the original file when `chmod --reference` is available. this ensures that when a file is marked web-accessible, it stays web-accessible when you run an in_place command on it.
Aaron Marcuse-Kubitza
02:22 PM Revision 8734: web/links/index.htm: BitTorrent Sync: security warning about data leak: added link to reported bug at http://forum.bittorrent.com/topic/18612-security-hole-in-linux-webui-makes-all-your-files-publicly-accessible/ . updated workaround instructions.
Aaron Marcuse-Kubitza
01:20 PM Revision 8733: web/links/index.htm: updated to Firefox bookmarks. BitTorrent Sync: added security warning that running btsync on Linux will by default create a *publicly-accessible, unprotected* WebUI, allowing anyone on the web to create a sync folder to view and edit any of your files. to avoid this, always run btsync with a config file which sets a password on the WebUI. if you have already run btsync, run `killall btsync` to turn off the WebUI.
Aaron Marcuse-Kubitza
12:59 PM Revision 8732: web/links/index.htm: updated to Firefox bookmarks. changes include links about BitTorrent Sync.
Aaron Marcuse-Kubitza
12:51 PM Revision 8731: lib/Firefox_bookmarks.reformat.csv: label <meta> description: renamed label text to "page's description" to be understandable to non-programmers
Aaron Marcuse-Kubitza
12:49 PM Revision 8730: lib/Firefox_bookmarks.reformat.csv: label <meta> description: remove any existing description label, to avoid duplicating the description label when the text of the description label changes
Aaron Marcuse-Kubitza
12:37 PM Revision 8729: lib/Firefox_bookmarks.reformat.csv: label <meta> description as such to avoid confusing it with comments added by the bookmark author, who may not agree with it. the <meta> description is in the 1st paragraph(s), when provided.
Aaron Marcuse-Kubitza
12:13 PM Revision 8728: lib/Firefox_bookmarks.reformat.csv: remove LAST_MODIFIED attributes, which seem to change whenever the favicon is updated and clutter up the svn diff
Aaron Marcuse-Kubitza
12:05 PM Revision 8727: lib/Firefox_bookmarks.reformat.csv: remove ICON attributes with favicon contents, which clutter up the svn diffs
Aaron Marcuse-Kubitza
11:52 AM Revision 8726: added web/links/index.htm.run with clean_up action to run whenever the index.htm bookmarks file is changed (`web/links/index.htm.run clean_up`)
Aaron Marcuse-Kubitza
11:51 AM Revision 8725: added lib/Firefox_bookmarks.reformat.csv
Aaron Marcuse-Kubitza

04/25/2013

09:58 PM Revision 8724: README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: exclude only backup files in the Documents/BIEN/svn/backups/ dir (sync the other files)
Aaron Marcuse-Kubitza
09:53 PM Revision 8723: README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: when changes are made on vegbiendev: don't sync the group, which can be bien on vegbiendev but not on jupiter (where aaronmk is not a member of the bien group)
Aaron Marcuse-Kubitza
09:51 PM Revision 8722: README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: when changes are made on vegbiendev: include *.md5
Aaron Marcuse-Kubitza
09:08 PM Revision 8721: README.TXT: Maintenance: removed step to backup the Ubuntu VM, because this is now done as part of the "backup files not in Time Machine" step
Aaron Marcuse-Kubitza
09:07 PM Revision 8720: README.TXT: Maintenance: added steps to backup files not in Time Machine
Aaron Marcuse-Kubitza
08:57 PM Revision 8719: README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: back up (almost) all files in the home directory instead of just Library/ (to both jupiter and Dropbox)
Aaron Marcuse-Kubitza
06:45 PM Revision 8718: README.TXT: put commands: removed unnecessary /** after dirs to exclude
Aaron Marcuse-Kubitza
06:26 PM Revision 8717: README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: added steps to also backup the settings to Dropbox (in addition to jupiter)
Aaron Marcuse-Kubitza
06:13 PM Revision 8716: lib/util.sh: added realpath () and use it where `readlink -f` is used
Aaron Marcuse-Kubitza
06:05 PM Revision 8715: lib/util.run: doc comment: changed "run scripts" to "runscripts"
Aaron Marcuse-Kubitza
06:04 PM Revision 8714: lib/util.run: factored general utils (non-runscript-specific) out into new util.sh
Aaron Marcuse-Kubitza
05:55 PM Revision 8713: lib/util.run: renamed $verbose to $verbosity because it's an integer level, not a boolean. continue to support specifying the verbosity in $verbose.
Aaron Marcuse-Kubitza
05:52 PM Revision 8712: lib/util.run: support $verbose="", which is treated as 0
Aaron Marcuse-Kubitza
05:50 PM Revision 8711: lib/util.run: only automatically echo env vars when they are set if verbosity is >= 2
Aaron Marcuse-Kubitza
05:49 PM Revision 8710: lib/util.run: added $verbose var to control logging. this is normally set to 0, but defaults to 2 for a runscript.
Aaron Marcuse-Kubitza
05:41 PM Revision 8709: lib/util.run: moved `shopt -s expand_aliases` inside the include guard so callers can override the setting without it being reset if util.run is included again
Aaron Marcuse-Kubitza
05:39 PM Revision 8708: lib/local.run: add $bin_dir to the PATH
Aaron Marcuse-Kubitza
05:15 PM Revision 8707: bin/in_place: don't run if no command to run, because that would erase the file
Aaron Marcuse-Kubitza
05:10 PM Revision 8706: bin/in_place: use -e (exit on error) option to sh instead of adding `|| exit` after every command
Aaron Marcuse-Kubitza
04:49 PM Revision 8705: *run: added include guards
Aaron Marcuse-Kubitza
04:30 PM Revision 8704: bugfix: lib/util.run: added include guard to prevent issues in "diamond includes", where a runscript includes two runscripts and *each* includes util.run, causing the second include of util.run to reset any methods overridden in the first included runscript. this prevents util.run from being included twice in the first place.
Aaron Marcuse-Kubitza
04:25 PM Revision 8703: lib/util.run: added include_guard_var (), self_not_included () for use by path-based include guards
Aaron Marcuse-Kubitza
03:40 PM Revision 8702: lib/util.run: echo_stdin (): print a separator before and after the echoed output to help distinguish it from other verbose output
Aaron Marcuse-Kubitza
03:37 PM Revision 8701: lib/table.run: use new local.run instead of defining $bin_dir, etc. itself
Aaron Marcuse-Kubitza
03:36 PM Revision 8700: lib/util.run: echo_export (): run the export builtin using `builtin` to avoid recursively calling any export alias or function defined by the user (e.g. `alias export=echo_export`. this causes util.run to be idempotent, such that it can be included multiple times without causing problems due to the redefinition of functions, vars, etc.
Aaron Marcuse-Kubitza
03:04 PM Revision 8699: added lib/local.run, which links to locally-available resources such as $bin_dir
Aaron Marcuse-Kubitza
02:50 PM Revision 8698: lib/table.run: added mysql (), mysql_ANSI ()
Aaron Marcuse-Kubitza
02:41 PM Revision 8697: added web/links/index.htm containing Firefox bookmarks exported to HTML. CSS style information has been added to fix default HTML tag formatting, improve readability, and handle the Firefox bookmarks format. <dd> tags have been manually closed where HTML would not auto-close them properly. note that these changes must be reapplied each time the bookmarks file is reexported (`svn di` will show the changes to make).
Aaron Marcuse-Kubitza
04:54 AM Revision 8696: lib/table.run: run mk_esc_name for $schema, $table
Aaron Marcuse-Kubitza

04/24/2013

11:18 PM Revision 8695: lib/table.run: added esc_name (), mk_esc_name ()
Aaron Marcuse-Kubitza
11:16 PM Revision 8694: lib/util.run: added set_var ()
Aaron Marcuse-Kubitza
10:58 PM Revision 8693: bugfix: lib/util.run: run_args_cmd (): `set --`: need to use eval so that the output of reverse is interpreted as escaped args rather than split using $IFS
Aaron Marcuse-Kubitza
10:40 PM Revision 8692: bugfix: lib/util.run: run_args_cmd (): BASH_ARGV is stored in reverse order, so it needs to be reversed to be usable as $@
Aaron Marcuse-Kubitza
10:38 PM Revision 8691: lib/util.run: added reverse ()
Aaron Marcuse-Kubitza
02:51 PM Revision 8690: schemas/VegCore/VegCore.ERD.mwb: regenerated exports
Aaron Marcuse-Kubitza
02:49 PM Revision 8689: schemas/VegCore/VegCore.ERD.mwb: source: renamed contact -> owner because smaller sources may not have an admin/contact person, but would have a copyright holder or other entity with control over the source's distribution
Aaron Marcuse-Kubitza
02:45 PM Revision 8688: schemas/VegCore/VegCore.ERD.mwb: added table comments from VegCore data dictionary (VegCore.vegpath.org)
Aaron Marcuse-Kubitza
01:08 PM Revision 8687: schemas/VegCore/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
01:07 PM Revision 8686: schemas/VegCore/VegCore.ERD.mwb: added hstore expansion columns where needed
Aaron Marcuse-Kubitza
12:54 PM Revision 8685: schemas/VegCore/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
12:34 PM Revision 8684: schemas/VegCore/VegCore.ERD.mwb: specimen: added unique constraint for each set of identifying columns
Aaron Marcuse-Kubitza
12:25 PM Revision 8683: schemas/VegCore/VegCore.ERD.mwb: collection: added back primary key
Aaron Marcuse-Kubitza
12:24 PM Revision 8682: schemas/VegCore/VegCore.ERD.mwb: specimen: made owner an fkey to collection instead of organization to correspond with orig_collection and current_collection (and to allow auto-filling it from current_collection in a trigger)
Aaron Marcuse-Kubitza
12:18 PM Revision 8681: schemas/VegCore/VegCore.ERD.mwb: specimen: reordered columns to group together each method of identifying a specimen
Aaron Marcuse-Kubitza
12:14 PM Revision 8680: schemas/VegCore/VegCore.ERD.mwb: collection: don't inherit from source, because a collection is a scoping entity, not a dataset. a collection *may* also be used as a dataset, but need not be for the purposes of scoping a specimen's accession_number.
Aaron Marcuse-Kubitza
12:07 PM Revision 8679: schemas/VegCore/VegCore.ERD.mwb: specimen: removed redundant stamping_institution and donor_institution, because these are now in orig_collection.institution. (it is assumed that stamping_institution and donor_institution are the same, because the first institution to acquire and store a specimen (the donor) is the one that stamps it with an accession_number.)
Aaron Marcuse-Kubitza
12:01 PM Revision 8678: schemas/VegCore/VegCore.ERD.mwb: specimen: split collection into orig,current_collection because the collection that scoped the accession_number (the orig_collection) is sometimes different than the collection where the specimen is now held (the current_collection, or specimen_holder_institutions), e.g. when the specimen is donated
Aaron Marcuse-Kubitza
11:49 AM Revision 8677: schemas/VegCore/VegCore.ERD.mwb: collection: added institution, name to identify and scope the collection
Aaron Marcuse-Kubitza
11:39 AM Revision 8676: schemas/VegCore/VegCore.ERD.mwb: changed taxon_name fkeys to taxon_concept where appropriate, and renamed the referencing columns to *_taxon_concept to clearly indicate the referenced table. taxon_concept fkeys are appropriate whenever a taxon is defined or associated with an observed entity, in which case the assertion is not just to a taxon name but to a defined taxon concept (albeit sometimes an unofficial taxon concept scoped by the collector or identifier).
Aaron Marcuse-Kubitza
11:08 AM Revision 8675: schemas/VegCore/VegCore.ERD.mwb: specimen: added code_in_individual column
Aaron Marcuse-Kubitza
10:40 AM Revision 8674: schemas/VegCore/VegCore.ERD.mwb: specimen: added fkey to individual for the individual the specimen was taken from
Aaron Marcuse-Kubitza
10:30 AM Revision 8673: schemas/VegCore/VegCore.ERD.mwb: individual/stem_observation: made individual/stem nullable because there is not always enough identifying information (stem tag, etc.) to actually associate an identifiable individual with an observation
Aaron Marcuse-Kubitza
10:27 AM Revision 8672: schemas/VegCore/VegCore.ERD.mwb: moved code from individual to individual_observation because the code is something applied at observation time, rather than something that can be used to track individuals *between observations*
Aaron Marcuse-Kubitza
10:17 AM Revision 8671: schemas/VegCore/VegCore.ERD.mwb: moved collection_event from taxon_occurrence to specimen because only specimens are actually collected, while other taxon_occurrences (e.g. individuals) must be uniquely identified by some means that are shared between re-observations (e.g. the stem tag)
Aaron Marcuse-Kubitza
10:03 AM Revision 8670: schemas/VegCore/VegCore.ERD.mwb: moved collector, collector_number from taxon_occurrence to taxon_observation because these are actually attributes that are available or assigned at observation time, rather than being intrinsic to the taxon_occurrence (which can be either an individual or a specimen)
Aaron Marcuse-Kubitza
09:34 AM Revision 8669: schemas/VegCore/VegCore.ERD.mwb: method: colored it forest green like stratum because these are both tables in the same category (method), and forest green is the color VegBank uses for methods. (the fact that VegBank gives stratum the method color indicates that we are on the right track when we are treating stratum as a type of method that's essentially a height class.)
Aaron Marcuse-Kubitza
09:27 AM Revision 8668: schemas/VegCore/VegCore.ERD.mwb: method: added parent fkey which can be used to autoinherit attributes from another method, in the same way that event autoinherits a method from its parent
Aaron Marcuse-Kubitza
09:16 AM Revision 8667: schemas/VegCore/VegCore.ERD.mwb: replaced < event label for diagram half with < place, because the occurrence half actually also includes events (observations), but does not include places
Aaron Marcuse-Kubitza
09:10 AM Revision 8666: schemas/VegCore/VegCore.ERD.mwb: removed extra empty page used for moving objects around
Aaron Marcuse-Kubitza
09:08 AM Revision 8665: schemas/VegCore/VegCore.ERD.mwb: swapped position of place and source categories so that the place tables, which users are apt to query often, are at the top, and the attribution information, which is likely to be used less often, is at the bottom. this also moves the plot table closer to taxon_occurrence, which should eventually have an fkey to it.
Aaron Marcuse-Kubitza
08:19 AM Revision 8664: schemas/VegCore/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
08:05 AM Revision 8663: schemas/VegCore/VegCore.ERD.mwb: record: added source_id_scope and added it to the record_unique unique index
Aaron Marcuse-Kubitza
08:01 AM Revision 8662: schemas/VegCore/VegCore.ERD.mwb: Default*ColumnType: updated to VARBINARY type used in key columns
Aaron Marcuse-Kubitza

04/23/2013

11:19 PM Revision 8661: schemas/VegCore/VegCore.ERD.mwb: TNRS taxonomic scrubbing steps: repositioned arrow labels so they point in the direction of the fkey lines
Aaron Marcuse-Kubitza
11:06 PM Revision 8660: schemas/VegCore/VegCore.ERD.mwb.run: use new zip/unzip_newer
Aaron Marcuse-Kubitza
11:06 PM Revision 8659: lib/util.run: added zip/unzip_newer aliases
Aaron Marcuse-Kubitza
11:05 PM Revision 8658: lib/util.run: added zip/unzip echo_run aliases
Aaron Marcuse-Kubitza
10:57 PM Revision 8657: schemas/VegCore/VegCore.ERD.mwb.run: export_ (): always overwrite existing files. this option is safe, because -u only extracts files with a newer mtime than their copies on disk.
Aaron Marcuse-Kubitza
10:55 PM Revision 8656: schemas/VegCore/VegCore.ERD.mwb: TNRS taxonomic scrubbing steps labels: converted tables to notes, which now have bold formatting so that empty tables aren't needed for formatting purposes
Aaron Marcuse-Kubitza
10:25 PM Revision 8655: schemas/VegCore/VegCore.ERD.mwb.run: import/export_ (): use zip/unzip's -u flag instead of `make` to auto-sync the zip file contents. the -u flag has the added benefit of preserving the contained file's mtime, at the expense of an mtime lookup in the zip archive's table of contents.
Aaron Marcuse-Kubitza
10:17 PM Revision 8654: added schemas/VegCore/VegCore.ERD.mwb.run, which syncs VegCore.ERD.mwb's zip file contents
Aaron Marcuse-Kubitza
09:58 PM Revision 8653: bugfix: lib/util.run: inline_make (): put echo_func at the top of the function
Aaron Marcuse-Kubitza
09:58 PM Revision 8652: lib/util.run: inline_make (): pass Makefile commands using /dev/fd/3 (i.e. 3<<'EOF' ...) instead of stdin, to allow stdin to be used by commands that prompt the user
Aaron Marcuse-Kubitza
09:44 PM Revision 8651: lib/util.run: inline_make (): use <() to pass Makefile commands as a named descriptor rather than using stdin, to allow stdin to be used by commands that prompt the user
Aaron Marcuse-Kubitza
09:27 PM Revision 8650: bugfix: lib/util.run: echo_vars (): remove any value to set (=*) from the vars, since this is often used with export
Aaron Marcuse-Kubitza
09:14 PM Revision 8649: bugfix: lib/util.run: echo_export (): set the vars before echoing them
Aaron Marcuse-Kubitza
08:55 PM Revision 8648: lib/util.run: run_args_cmd (): use new canon_rel_path() to resolve .. in the script path, so it's easier to see where the script is actually located
Aaron Marcuse-Kubitza
08:52 PM Revision 8647: lib/util.run: echo_func (): use new canon_rel_path() to resolve .. in the script path, so it's easier to see where the script is actually located
Aaron Marcuse-Kubitza
08:51 PM Revision 8646: lib/util.run: added canon_rel_path ()
Aaron Marcuse-Kubitza
08:40 PM Revision 8645: lib/util.run: inline_make (): provide $self as make var
Aaron Marcuse-Kubitza
08:35 PM Revision 8644: lib/util.run: inline_make (): provide self_dir to make as env var instead of as make var, to avoid specifying every make var both as a local var and as a make var in the make command
Aaron Marcuse-Kubitza
08:31 PM Revision 8643: lib/util.run: export alias: added comment explaining what it does
Aaron Marcuse-Kubitza
08:30 PM Revision 8642: lib/util.run: aliased export to echo_export so all env vars are automatically echoed when they are set
Aaron Marcuse-Kubitza
08:29 PM Revision 8641: lib/util.run: added echo_vars (), echo_export ()
Aaron Marcuse-Kubitza
08:17 PM Revision 8640: lib/util.run: inline_make (): added local $self and use it to create $self_dir
Aaron Marcuse-Kubitza
08:15 PM Revision 8639: lib/util.run: inline_make (): set $self_dir as a local var before passing it to make, to avoid clutter in the command that invokes make
Aaron Marcuse-Kubitza
07:54 PM Revision 8638: added schemas/VegCore/document.mwb.xml, so changes to the diagram can be tracked line-by-line. keeping this up-to-date requires running `unzip VegCore.ERD.mwb document.mwb.xml` whenever VegCore.ERD.mwb is changed.
Aaron Marcuse-Kubitza
07:53 PM Revision 8637: schemas/VegCore/VegCore.ERD.mwb: target MySQL version: updated to 5.5.3
Aaron Marcuse-Kubitza
07:49 PM Revision 8636: schemas/VegCore/VegCore.ERD.mwb: NoteFigure:TextFont (labels): made it bold to stand out better
Aaron Marcuse-Kubitza
07:45 PM Revision 8635: schemas/VegCore/VegCore.ERD.mwb: Layer:TitleFont (legends): increased size and made it bold to stand out better
Aaron Marcuse-Kubitza
07:31 PM Revision 8634: schemas/VegCore/VegCore.ERD.mwb: made stratum extend method, so that any observation event can have an associated stratum. removed no longer needed plot_stratum, since strata are now represented as sampling methods (i.e. using height classes) rather than a variant of subplots.
Aaron Marcuse-Kubitza
07:21 PM Revision 8633: schemas/VegCore/VegCore.ERD.mwb: VegCore DB: changed collation to utf8_bin to match table collation
Aaron Marcuse-Kubitza
07:15 PM Revision 8632: schemas/VegCore/VegCore.ERD.mwb: saved settings for publishing schema to vegbiendev phpMyAdmin (vegbiendev.nceas.ucsb.edu/phpmyadmin/index.php?db=VegCore). this adds `DROP TABLE IF EXISTS` before each CREATE TABLE statement, renames the default DB to VegCore, and adds a USE statement to select the VegCore DB.
Aaron Marcuse-Kubitza
06:29 PM Revision 8631: schemas/VegCore/connections.xml: vegbiendev.nceas.ucsb.edu: set schema (DB) to VegCore
Aaron Marcuse-Kubitza
06:20 PM Revision 8630: added schemas/VegCore/connections.xml, which should be installed as the connections.xml settings file
Aaron Marcuse-Kubitza
06:19 PM Revision 8629: schemas/VegCore/VegCore.ERD.mwb: imported database connections from the connections.xml settings file into document.mwb.xml in the .mwb zip file. these connections are not used directly by MySQL Workbench, but instead need to be copied into the connections.xml settings file.
Aaron Marcuse-Kubitza
05:17 PM Revision 8628: schemas/VegCore/VegCore.ERD.mwb: use Database > Forward Engineer (Ctrl+G) instead of File > Export > Forward Engineer SQL CREATE Script (Ctrl+Shift+G), so that comments will use the max comment length for the *server's* MySQL version rather than MySQL Workbench's internal MySQL version, which has a max comment length of only 60 chars for tables. this commit stores the default export settings for Forward Engineer, which are saved each time it's run.
Aaron Marcuse-Kubitza
04:31 PM Revision 8627: bugfix: root Makefile: $(postgresReload-Linux): fixed the "shmat(id=...) failed: Cannot allocate memory" error by telling the user to restart PostgreSQL manually from the command line
Aaron Marcuse-Kubitza
04:29 PM Revision 8626: root Makefile: $(postgresReload-Linux): don't restart Postgres twice, because this does not fix the "shmat(id=...) failed: Cannot allocate memory" error. this error is instead produced whenever PostgreSQL is restarted from *within a Makefile*, but not when it's run directly from the command line or /bin/sh.
Aaron Marcuse-Kubitza
03:20 PM Revision 8625: root Makefile: $(postgresReload-Linux): restart Postgres twice, because the first time after the postgresql.conf has been udpated, it always produces a "shmat(id=...) failed: Cannot allocate memory" error
Aaron Marcuse-Kubitza
03:10 PM Revision 8624: schemas/VegCore/VegCore.ERD.mwb: re-synced with VegCore.my.sql: step 2: `DEFAULT NULL` added on all nullable columns
Aaron Marcuse-Kubitza
03:10 PM Revision 8623: schemas/VegCore/VegCore.ERD.mwb: re-synced with VegCore.my.sql: step 1: table order changed
Aaron Marcuse-Kubitza
02:54 PM Revision 8622: schemas/VegCore/VegCore.ERD.mwb: relationship: renamed the two fk_relationship_record1 fkeys so their names wouldn't collide
Aaron Marcuse-Kubitza
02:52 PM Revision 8621: schemas/VegCore/VegCore.ERD.mwb: changed encoding back to utf8/utf8_bin now that latin1 isn't needed to make the key columns satisfy the index length limits. note that the connection encoding must be utf8 on import so that the correct byte sequence is stored in the VARBINARY columns.
Aaron Marcuse-Kubitza
02:11 PM Revision 8620: schemas/VegCore/VegCore.ERD.mwb: changed VARCHAR BINARY columns to VARBINARY because VARCHAR BINARY is actually only an alias for the *_bin collation of the table's encoding, but still uses multibyte characters if the encoding does. note that VARBINARY will still be displayed as text (in the connection encoding), rather than hex values.
Aaron Marcuse-Kubitza
02:02 PM Revision 8619: schemas/VegCore/VegCore.ERD.mwb: changed the type of TEXT columns used in indexes to VARCHAR(767) BINARY to satisfy prefix length limits ("a prefix can be up to [...] 767 bytes for InnoDB tables" <http://dev.mysql.com/doc/refman/5.5/en/create-index.html>). the BINARY flag was added to allow storing Unicode values (from UTF-8), which are not representable in the latin1 encoding.
Aaron Marcuse-Kubitza
01:15 PM Revision 8618: schemas/VegCore/VegCore.ERD.mwb: changed encoding to latin1 to fit more characters in length-limited indexes (indexes treat utf8 as having *3* bytes/character (http://stackoverflow.com/questions/1814532/1071-specified-key-was-too-long-max-key-length-is-767-bytes#comment-18031137), which significantly reduces the max length of an indexed column)
Aaron Marcuse-Kubitza
01:08 PM Revision 8617: schemas/VegCore/VegCore.ERD.mwb: changed collation to utf8_bin to avoid case-*in*sensitive comparison of case-sensitive strings. (utf8_bin is also much faster than encoding-aware collations because it uses the raw bytes.)
Aaron Marcuse-Kubitza
12:47 PM Revision 8616: schemas/VegCore/VegCore.ERD.mwb: set DefaultPkColumnType to TEXT to match the type of current pkey columns
Aaron Marcuse-Kubitza
12:16 PM Revision 8615: schemas/VegCore/VegCore.ERD.mwb: set all tables to use utf8/utf8_unicode_ci rather than latin1/latin1_swedish_ci because our DB is in UTF-8. utf8_unicode_ci is better than utf8_general_ci because it supports international characters better (http://forums.mysql.com/read.php?103,187048,188748)
Aaron Marcuse-Kubitza
12:10 PM Revision 8614: schemas/VegCore/VegCore.ERD.mwb: TNRS taxonomic scrubbing steps labels: removed fkey columns to the next step since these are not needed and were causing problems with round-trip sync to the .my.sql file
Aaron Marcuse-Kubitza
11:42 AM Revision 8613: bugfix: schemas/VegCore/VegCore.ERD.mwb: fixed foreign keys without a referenced columns list
Aaron Marcuse-Kubitza
11:36 AM Revision 8612: schemas/VegCore/VegCore.ERD.mwb: upgraded for MySQL Workbench 5.2.47 (MySQL 5.5), which requires indexes to have different names than fkeys
Aaron Marcuse-Kubitza
10:25 AM Revision 8611: schemas/postgresql.conf: shared_buffers: reduced again for Ubuntu VM
Aaron Marcuse-Kubitza
10:22 AM Revision 8610: bugfix: schemas/postgresql.conf: shared_buffers: removed invalid Unicode character U+FEFF
Aaron Marcuse-Kubitza
02:40 AM Revision 8609: schemas/postgresql.conf: shared_buffers: reduced to work in Ubuntu VM
Aaron Marcuse-Kubitza
12:58 AM Revision 8608: README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: add * after analytical_aggregate.*.csv to match *.md5 files as well
Aaron Marcuse-Kubitza

04/19/2013

07:33 PM Revision 8607: inputs/VegBIEN/TWiki/templates/login.tmpl: increased font size further using <span> style. changed text to fit on two lines with the new size.
Aaron Marcuse-Kubitza
07:26 PM Revision 8606: inputs/VegBIEN/TWiki/templates/login.tmpl: added large text with instructions to register (the TWiki instructions are small and easily missed)
Aaron Marcuse-Kubitza
07:09 PM Revision 8605: added inputs/VegBIEN/TWiki/templates/login.tmpl
Aaron Marcuse-Kubitza
05:14 PM Revision 8604: inputs/VegBIEN/TWiki/twiki.conf: main TWiki dir: changed redirect to go directly to the Main TWiki web, so that we don't need to worry about creating different "webs" (subsites) for top-level wiki pages
Aaron Marcuse-Kubitza
05:03 PM Revision 8603: inputs/VegBIEN/TWiki/twiki.conf: main TWiki dir: redirect shortened wiki page URLs of the form vegpath.org/twiki/Path to do/view/ for use with VegPath
Aaron Marcuse-Kubitza
04:49 PM Revision 8602: added web/.twiki symlink to datasources/VegBIEN/TWiki/
Aaron Marcuse-Kubitza
04:48 PM Revision 8601: added inputs/VegBIEN/TWiki/twiki.conf
Aaron Marcuse-Kubitza
12:58 PM Revision 8600: schemas/VegCore/VegCore.ERD.mwb: coordinates: added latitude_deg, longitude_deg. geovalidation: added lat_long_in_ranks hstore.
Aaron Marcuse-Kubitza
12:32 PM Revision 8599: schemas/VegCore/VegCore.ERD.mwb: realigned event/occurrence divider. moved method to avoid overlapping the halves.
Aaron Marcuse-Kubitza
12:29 PM Revision 8598: schemas/VegCore/VegCore.ERD.mwb: specimen: replaced specimen_holder_institutions with specimen_holder_institution many:many table. added fkeys for the various institution fields. collection: removed name because this is already a required field in source. event_participant: removed *_id suffixes from fields.
Aaron Marcuse-Kubitza
11:55 AM Revision 8597: schemas/VegCore/VegCore.ERD.mwb: replaced party_list with event_participant many:many table, since the nature of each type of party_list (ordered, not, etc.) may differ between tables that HAVE-A party_list, and because it is not possible to create an fkey to a column that is not unique in the referenced table (as is the case for party_list.list_id)
Aaron Marcuse-Kubitza
11:41 AM Revision 8596: schemas/VegCore/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
11:28 AM Revision 8595: schemas/VegCore/VegCore.ERD.mwb: made first_publisher a field of source (fkey to party) instead of a subclass of it, because a source can be both an aggregator for some datasets and a first publisher for others and which one it is depends on the dataset
Aaron Marcuse-Kubitza
11:06 AM Revision 8594: schemas/VegCore/VegCore.ERD.mwb: added organization
Aaron Marcuse-Kubitza
11:01 AM Revision 8593: schemas/VegCore/VegCore.ERD.mwb: added party_list
Aaron Marcuse-Kubitza
10:29 AM Revision 8592: schemas/VegCore/VegCore.ERD.mwb: regenerated exports
Aaron Marcuse-Kubitza
10:09 AM Revision 8591: schemas/VegCore/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
09:50 AM Revision 8590: schemas/VegCore/VegCore.ERD.mwb: taxon_presence, aggregate_observation: added required taxon field to identify (distinguish) the observation itself. this is expected to be the same as the first taxon_determination applied to the observation.
Aaron Marcuse-Kubitza
09:21 AM Revision 8589: schemas/VegCore/VegCore.ERD.mwb: collection: renamed code to name because the collection code is often the full name of the collection, rather than just an abbreviation. made name required.
Aaron Marcuse-Kubitza
09:17 AM Revision 8588: schemas/VegCore/VegCore.ERD.mwb: source: added name
Aaron Marcuse-Kubitza

04/18/2013

04:54 PM Revision 8587: schemas/VegCore/VegCore.ERD.mwb: set page size to fit the ERD contents. this cannot be done through the GUI, so instead you need to use the following steps:
$ unzip ERD.mwb document.mwb.xml
open document.mwb.xml and search for "this ERD"
units are mm
$ zip -u ERD.mwb docume...
Aaron Marcuse-Kubitza
 

Also available in: Atom