Project

General

Profile

Statistics
| Revision:

# Date Author Comment
9417 05/16/2013 04:38 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: $(svnFiles): also exclude *.data.sql, which should never be in svn

9416 05/16/2013 04:27 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: cultivated_family_locations: documented that table is from sftp://nimoy.nceas.ucsb.edu/home/bien/bien2_scripts/geoscrub/cultivated/cult_by_taxon/flag_by_taxa.inc (i.e. not generated by a function)

9415 05/16/2013 04:15 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: place.geovalid: added latLongDomainValid to the values to _and together

9414 05/16/2013 04:09 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: place.geovalid: require it to be NOT NULL so that it's always a 2-valued boolean (but default it to false since it's not a required field)

9413 05/16/2013 04:06 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: place.geovalid: use false instead of NULL

9412 05/16/2013 03:46 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): exclude deleted rows (i.e. where the deleted timestamp is non-NULL)

9411 05/16/2013 03:42 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/header.csv: regenerated using ./run. since the table is reimported as a CSV, it uses bin/csv2db, which prepends an additional row_num column.

9410 05/16/2013 03:09 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): remove explicit cols list to include all cols. the file size of the generated table.tsv will increase by ~3x, but should remain reasonably-sized compared to our available disk space.

9409 05/16/2013 03:04 PM Aaron Marcuse-Kubitza

bugfix: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): need \ line continuation after vars so they only apply to the command rather than being set as global vars

9408 05/16/2013 03:02 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: load_data(): use new $verbosity_min instead of running `verbosity_min` so that the command name logging is not output with the new verbosity

9407 05/16/2013 02:59 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added $verbosity_min to set a `verbosity_min` value after the command name, etc. has been logged, so that the logging itself is not output with the new verbosity

9406 05/16/2013 02:38 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: range_modeling_input: include only plants (i.e. rows with higher_plant_group IS NOT NULL)

9405 05/16/2013 02:36 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: range_modeling_input: added higher_plant_group, for use in restricting rows to plants

9404 05/16/2013 02:24 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/map.csv: *validity: added definitions of the numeric codes from _src/README.TXT

9403 05/16/2013 01:11 PM Aaron Marcuse-Kubitza

added planning/workflow/validation/GeoDistKM.sql.txt

9402 05/16/2013 12:42 PM Aaron Marcuse-Kubitza

planning/goals/BIEN3_derived_data_products.docx: updated to most recent version from Brad's e-mail on 2013-4-16

9401 05/16/2013 11:18 AM Aaron Marcuse-Kubitza

/README.TXT: Full database import: before running screen: added `unset TMOUT` because TMOUT (autologout) causes screen to exit even with background processes active

9400 05/16/2013 11:17 AM Aaron Marcuse-Kubitza

/README.TXT: Maintenance: added things to put in your .profile on a live machine (e.g. vegbiendev). in particular, you MUST NOT have a TMOUT (autologout) set, because this causes screen to exit even if background processes (e.g. from column-based import) are running

9399 05/16/2013 11:15 AM Aaron Marcuse-Kubitza

/README.TXT: Maintenance: added things to put in your .profile on a live machine (e.g. vegbiendev). in particular, you MUST NOT have a TMOUT (autologout) set, because this causes screen to exit even if background processes (e.g. from column-based import) are running

9398 05/16/2013 06:15 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: load_data(): need to ensure the verbosity is at least 3 because the install logs require verbose output. (3 is the default for the installer, but is overridden by the runscripts, which instead set the default to 1.)

9397 05/16/2013 06:13 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: added verbosity_min()

9396 05/16/2013 06:00 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: added verbosity_int() and use it instead of `round_down "$verbosity"`

9395 05/16/2013 02:08 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: log+ alias: don't expand next word because it's not a cmd

9394 05/16/2013 01:59 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: $schema, $table: log the `cd` used to calculate the value at log_level 3 instead of 1 (note that the cd() function call for this will be logged at log_level 5)

9393 05/16/2013 01:51 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: cd(): trace-log the cd() function call itself (at log_level 3) in addition to the cd builtin call

9392 05/15/2013 11:28 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: import(): added step to load the data into the staging table before postprocessing it

9391 05/15/2013 11:27 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: moved table.tsv.md5/make() and invocation of it to inputs/GBIF/table.run because it's general to all tables (which would all use table.tsv for this datasource). use $target_filename in calling table.tsv.md5/make from table.tsv/make.

9390 05/15/2013 11:21 PM Aaron Marcuse-Kubitza

bugfix: lib/sql.py: parse_exception(): typed_name_re: need to ensure that full name is matched rather than just first character

9389 05/15/2013 10:26 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. Mountain Lion upgrade: added Python psycopg2, Python OrderedDict, X11.

9388 05/15/2013 07:51 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: table_make(): run canon_rel_path on the datasrc dir so it's displayed as the direct path to it, without ..

9387 05/15/2013 07:47 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: table_make(): moved comment about "${@/#/$table/}" to right after the line it describes

9386 05/15/2013 07:44 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): renamed to table_make() to make it clear that the target names are relative to the table subdir itself, not the datasrc dir. it was previously called input_make because it used inputs/input.Makefile directly, but now will use any Makefile in the datasrc dir.

9385 05/15/2013 06:44 PM Aaron Marcuse-Kubitza

added inputs/GBIF/Makefile, which links to ../input.Makefile, to allow running make directly in the datasrc dir (i.e. without --makefile=.../input.Makefile). this is required by the runscripts.

9384 05/15/2013 03:34 PM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv.md5/make(): don't add extra .md5 extension to $target_filename because it already has the extension as part of the target name (now that this command is run in its own make target rather than in table.tsv/make())

9383 05/15/2013 03:30 PM Aaron Marcuse-Kubitza

lib/sql_gen.py: import OrderedDict from collections instead of ordereddict for Mac 10.8 Mountain Lion upgrade

9382 05/15/2013 02:29 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: renamed $log_level_indent to $log_indent_step to avoid confusion with the log_level, which is a different kind of indent (using + signs instead of |s)

9381 05/15/2013 02:26 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: also export PS4, because it follows verbosity and therefore also needs to be propagated to invoked commands

9380 05/15/2013 02:21 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: can_log(): support decimal verbosities using round_down()

9379 05/15/2013 02:21 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: always use echo_export instead of export, even when the verbosity at load time would suppress output, because the verbosity may actually increase during the script due to log-- calls, etc., and vars should then still be echoed as expected

9378 05/15/2013 02:17 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: log+: inlined PS4_prefix_n alias because there is now room for it

9377 05/15/2013 02:16 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: log+: $verbosity: support decimal verbosities (but not decimal log_levels) by using new float+int

9376 05/15/2013 02:13 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: removed no longer used log-. use log+ with a negative argument instead.

9375 05/15/2013 02:12 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: log+: PS4 when $1 < 0: need to negate $1 because now it's a negative number

9374 05/15/2013 02:06 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: log--: use log+ with 1 instead of log so we don't need a separate log- function

9373 05/15/2013 02:04 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: log+: support negative log_level adjustments. log-: use log+ with the negative of its argument

9372 05/15/2013 01:48 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. sorted MySQL and PostgreSQL sections.

9371 05/15/2013 01:42 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks

9370 05/15/2013 01:39 PM Aaron Marcuse-Kubitza

web/links/index.htm: Mac 10.8 Mountain Lion > PostgreSQL: appended tab to name to disambiguate it from the general PostgreSQL section

9369 05/15/2013 01:33 PM Aaron Marcuse-Kubitza

bugfix: web/links/index.htm.run: prepend __ to the HTML anchors of bookmarks toolbar links so they don't shadow/override folders of the same name: also renamed the corresponding self-anchor hyperlinks

9368 05/15/2013 01:32 PM Aaron Marcuse-Kubitza

bugfix: web/links/index.htm.run: prepend __ to the HTML anchors of bookmarks toolbar links so they don't shadow/override folders of the same name: need to replace all occurrences (using /g option to sed) to include both HTML anchors on the line

9367 05/15/2013 01:26 PM Aaron Marcuse-Kubitza

web/links/index.htm.run: prepend __ to the HTML anchors of bookmarks toolbar links so they don't shadow/override folders of the same name

9366 05/15/2013 01:23 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: sed: changed it to an alias so it will also be expanded when passed to an external command (like in_place) that can only run an executable, not a shell function (this occurs as long as the external command is defined as an alias which ends in space, to alias-expand the next word). added associated $sed_cmd var for cases when there is no alias wrapper around the external command, and the literal alias body must be used instead.

9365 05/15/2013 11:02 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: log+/-(): setting verbosity: added space around operators to support negative numbers

9364 05/15/2013 10:55 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added float functions (esp. float+int()) for dealing with decimal verbosities used by sql.py and column-based import

9363 05/15/2013 10:53 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: inputs/GBIF/raw_occurrence_record/run: added check_target_exists so you know why make skipped the file (for other, non-silent targets, it would also avoid make's verbose output when the file exists)

9362 05/15/2013 10:38 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): moved making of table.tsv.md5 to separate function

9361 05/15/2013 10:30 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: $log_fd: indicated that this is only initially stderr (however, the new port will just use stderr if it's not redirected separately)

9360 05/15/2013 10:28 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: verbosities: 3: added that this includes values of kw params

9359 05/15/2013 10:25 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: logging: added description of the verbosities available, including what each one does and what it's useful for:

  1. verbosities:
  2. 0: just print errors. useful for cron jobs.
  3. 1: also external commands run. useful for running at the command line....
9358 05/15/2013 10:11 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: command__set_fds(): echo the >& line at a higher log_level, because this information (i.e. which fd is used by the command for logging) is primarily for debugging and should not normally be printed

9357 05/15/2013 10:00 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): also add md5 sum for table.tsv

9356 05/15/2013 09:50 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added back filter kw args, which had gotten deleted in a commit without update (although actually, svn should not allow a commit without update, so the working copy may have gotten corrupted)

9355 05/15/2013 09:27 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): added $silent flag which turns on make's --silent option

9354 05/15/2013 09:25 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: set_make_vars: usage: added "use $target" to indicate that vars are made available by this alias

9353 05/15/2013 09:24 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: inline_make(): take the script stdin from caller-provided stdin and get make's stdin from global stdin, so that the caller can just use <<'EOF' rather than having to include a specific fd before the <<

9352 05/15/2013 08:46 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): documented that it requires a Makefile in the datasrc dir containing `include ../input.Makefile

9351 05/15/2013 08:41 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): added comment explaining that "${@/#/$table/}" replaces the empty str at the beginning of str (/#) with $table/

9350 05/15/2013 08:38 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: put functions on one line where possible

9349 05/15/2013 08:35 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): use the local make() function instead of external make directly, because it sets $cmd_log_fd appropriately to ensure that all the echoed make commands get properly logged to stdlog/stderr (stdlog is fd 30 when it's redirected to a file)

9348 05/15/2013 08:22 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run table.tsv/make() and functions used by it: added usage comments for cmd line usage, caller usage, and declaring function usage

9347 05/15/2013 07:29 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks

9346 05/15/2013 07:27 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks

9345 05/15/2013 07:00 AM Aaron Marcuse-Kubitza

web/links/index.htm: removed no longer needed - at the beginning of every folder's description

9344 05/15/2013 07:00 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: label page's description: don't do this for folders (i.e. descriptions preceded by an <H3> tag) because their descriptions are always author-added rather than from a web page. this avoids needing to add a - at the beginning of every folder's description.

9343 05/14/2013 09:43 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: put fixes subdirs in order.

9342 05/14/2013 09:42 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: put fixes subdirs in order.

9341 05/14/2013 08:56 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. upgrading Mac OS X to 10.8 Mountain Lion: added fixes for Apache ~/Sites dirs, Apache PHP

9340 05/14/2013 05:55 PM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. added bookmarks for upgrading Mac OS X to 10.8 Mountain Lion. (WARNING: DO NOT upgrade unless you are prepared to fix several programs broken by the upgrade: svn, PostgreSQL. instructions are in the corresponding bookmark subdirs. these programs will be COMPLETELY UNAVAILABLE until they are manually fixed!)

9339 05/13/2013 07:43 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): cols: also include scientific_name, which is preferable as a TNRS input because it also contains lower ranks

9338 05/13/2013 07:40 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): cols: also include id, institution_code, collection_code, catalogue_number

9337 05/13/2013 07:38 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added filter for institution_codes in herbaria.ih (in PostgreSQL)

9336 05/13/2013 07:19 AM Aaron Marcuse-Kubitza

inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added column subset (from http://vegpath.org/twiki/bin/view/Main/ConfCall20130509#subsetting_strategy > include)

9335 05/13/2013 07:14 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: mk_select: constructed queries: in ${var:+if_true} syntax, put the newline at the end of the if_true value instead of the beginning, so that each ${var:+if_true} expression starts at the beginning of a line

9334 05/13/2013 07:10 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: mk_select: constructed queries: support custom columns list using $cols

9333 05/13/2013 07:07 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: mk_select: constructed queries: support WHERE clause using $filter

9332 05/13/2013 06:52 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_stdin(): don' increase the log_level, because the input being sent to the command (which is usually a set of interpreted commands itself) is necessary to fully know what action is being performed

9331 05/13/2013 06:30 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: add a manual errexit for $() exprs by embedding them in just a var assignment (without local or declare), whose exit status will then equal the of the $(). a `|| return` also needs to be added because errexit does not work on assignment statements. this commit adds them for func_loc(), echo_func(), canon_rel_path(), set_paths(), save_cache, cached realpath(), local.sh global vars

9330 05/13/2013 06:14 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: shell-variable-based caching: usage: updated alias names

9329 05/13/2013 05:33 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. regenerating also removed extra <a name=...> tags that were added when running index.htm.run on an already-processed index.htm.

9328 05/13/2013 05:19 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: `set -o`: need a -o before every option to set

9327 05/13/2013 05:10 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' names: hyperlink the added anchors as clickable paragraph marks (like Redmine), which take you to the HTML anchor. this is analogous to the clickable folder names which take you to their anchors.

9326 05/13/2013 04:52 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' names, in addition to their URLs

9325 05/13/2013 04:47 AM Aaron Marcuse-Kubitza

bugfix: lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links' URLs: use .*? instead of .* to match the contents of the <A> tag before the HREF

9324 05/13/2013 04:45 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links using the URL itself as the anchor. these can be used to link to the comments attached to a bookmark in the bookmarks page, rather than to the bookmark's destination.

9323 05/13/2013 04:43 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: add HTML anchors for external links using the URL itself as the anchor. these can be used to link to the comments attached to a bookmark in the bookmarks page, rather than to the bookmark's destination.

9322 05/13/2013 04:35 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks

9321 05/13/2013 04:24 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: `set -o`: added pipefail option, to ensure that exit statuses (esp. for errexit) also work with pipelines (a|b)

9320 05/13/2013 04:13 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: use new log+ with a numeric arg instead of multiple calls to log++

9319 05/13/2013 04:06 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: use new log+ with a numeric arg instead of multiple calls to log++. use the command form of log-- (`log-- echo_func`) to counter the normal log++ performed by echo_func, for cases when the function name is descriptive and should be output at the same log_level as the commands it runs.

9318 05/12/2013 08:33 PM Aaron Marcuse-Kubitza

*{.sh,run}: removed extra space between function name and ()