Project

General

Profile

  • svn:executable: *

# Date Author Comment
13344 04/29/2014 09:38 PM Aaron Marcuse-Kubitza

fix: *{.sh,run}: stderr_matches() callers: added benign_error=1 where needed

13058 04/07/2014 01:43 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: table_make_install(): need to ignore skip_table() errexit

12968 03/29/2014 04:06 AM Aaron Marcuse-Kubitza

*{.sh,run}: runscript targets: use begin_target instead of echo_func so the target name is properly echoed. note that this requires using with_rm so that $rm is properly progagated to applicable invoked targets. (previously, $rm was progagated to all invoked targets. note that with_rm only works inside a runscript target that starts with begin_target.)

12967 03/29/2014 03:58 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: self_make(): renamed to with_rm() for clarity, since this is used only to progagate $rm, and does not also invoke a command with the same name as the current function, as the name might suggest

12878 03/24/2014 01:43 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: schema/make calls: need to use `make schema` instead because old-style datasources don't have a top-level runscript (the absence of this identifies them as old-style so inputs/input.Makefile works correctly)

12864 03/22/2014 04:44 AM Aaron Marcuse-Kubitza

fix: lib/runscripts/table.run: table_make_install(): need to inform the user when it skips installing a table, because this is often unexpected

12862 03/22/2014 04:15 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: removed no longer used datasrc_make_install()

12850 03/22/2014 02:59 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: load_data(): use the much simpler `schema/make` run target, rather than outsourcing to the legacy Makefile via the convoluted datasrc_make_install()/table_make_install()

12847 03/22/2014 02:14 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: psql: always include ; at end of statement

12804 03/21/2014 04:31 PM Aaron Marcuse-Kubitza

*{.sh,run}: stderr_matches() wrapper calls: removed no longer needed prep_try/rethrow

12800 03/21/2014 03:42 PM Aaron Marcuse-Kubitza

*{.sh,run}: stderr_matches() wrapper calls: removed no longer needed prep_try/rethrow

12797 03/21/2014 03:10 PM Aaron Marcuse-Kubitza

fix: lib/sh/db.sh: pg_table_exists(): use stderr_matches() rather than just the exit status. this also avoids highlighting the benign error.

12779 03/20/2014 07:58 PM Aaron Marcuse-Kubitza

*{.sh,run}: use new begin_target instead of `echo_func; set_make_vars`

12773 03/20/2014 07:13 PM Aaron Marcuse-Kubitza

bugfix: *{.sh,run}: stderr_matches() wrapper callers: use the required wrapper caller usage, which now includes rethrow and prep_try

12743 03/18/2014 04:04 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: custom_postprocess(): need to apply renames to SQL statements in postprocess.sql before it can be run

12691 03/14/2014 03:03 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: remake_VegBIEN_mappings(): renamed to just mappings() since action make targets should be short names

11970 01/20/2014 11:33 AM Aaron Marcuse-Kubitza

moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).

11913 12/16/2013 04:45 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: added test_() target and use it in remake_VegBIEN_mappings() (it would not be clear that remake_VegBIEN_mappings() runs the tests)

11770 11/26/2013 12:47 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: table_make_install(): simplified the setting of $noclobber since there no longer needs to be a different command for when the log exists

11769 11/26/2013 12:08 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: need to errexit the make target, so that errors in the SQL install scripts are not suppressed. this requires pre-checking if the table exists (using new pg_table_exists), so that the install target's errexit does not then need to be suppressed for cases when the table already exists.

11602 11/08/2013 10:51 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: reordered functions in the order they are called by import()

11252 10/12/2013 12:12 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: load_data(): pass $is_view through to `make reinstall` so that DROP VIEW will be used instead of DROP TABLE where applicable

11247 10/10/2013 05:46 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: load_data(): in remaking mode, need to remake header.csv in case the columns have changed

11000 09/17/2013 10:05 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/*: calls to rm: use `rm -f` instead to avoid an error (which aborts the program) if the file does not yet exist

10372 07/20/2013 03:39 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: postprocess(): added remake action that calls trim_table()

10371 07/20/2013 03:37 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: added trim_table(), which calls util.trim(regclass, regclass)

10370 07/20/2013 03:23 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: map_table(): added remake action that calls reset_col_names()

10369 07/20/2013 03:21 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: added reset_col_names(), which calls util.reset_col_names()

10368 07/20/2013 03:19 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: map_table(): moved $map_table to global var so it can be used by other functions

10367 07/20/2013 03:09 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: postprocess(): don't propagate $remake to remake_VegBIEN_mappings(), since this will cause map.csv to be remade, which is not related to the postprocessing.

10366 07/20/2013 03:08 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: map_table(): util.set_col_names_with_metadata(): removed unnecessary cast to regclass, which is performed implicitly. this used to be needed when the polymorphic util.rename_cols() was used instead.

10276 07/14/2013 02:14 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: postprocess(): propagate the $remake flag to remake_VegBIEN_mappings using self_make, so that a remake=1 on postprocess will cause map.csv to be regenerated as it would for a remake=1 directly on remake_VegBIEN_mappings

10275 07/14/2013 02:10 AM Aaron Marcuse-Kubitza

bugfix: postprocess(): moved $can_test flag from import() to this function because it is used here

10274 07/14/2013 02:08 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: import(): moved postprocessing commands to separate postprocess() function that can be invoked on an already-imported staging table to avoid running the load_data() target. this is especially useful when running the postprocessing on a working copy without the unversioned data files, for datasources whose load_data() target would otherwise try to download the files because they don't already exist.

10273 07/14/2013 02:01 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: postprocess(): renamed to custom_postprocess() since this runs only the datasource's custom postprocessing commands, not all the postprocessing commands including map_table, mk_derived

10252 07/11/2013 03:56 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: remake_VegBIEN_mappings(): when remaking, do not remake header.csv, because it should keep the original CSV columns rather than being reset to whatever the current staging table columns happen to be. to force-regenerate this, instead delete it first and then run remake_VegBIEN_mappings(). remake mode will now just regenerate map.csv from header.csv, in case map.csv's columns are incomplete or out of order.

10251 07/11/2013 03:55 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: remake_VegBIEN_mappings(): when remaking, do not remake header.csv, because it should keep the original CSV columns rather than being reset to whatever the current staging table columns happen to be. to force-regenerate this, instead delete it first and then run remake_VegBIEN_mappings(). remake mode will now just regenerate map.csv from header.csv, in case map.csv's columns are incomplete or out of order.

10250 07/11/2013 03:50 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: map_table(): do not rename view columns, since their column names come from their (column-renamed) joined tables rather than from a map.csv. header.csv, map.csv for views will generally become out-of-date whenever the joined tables change, so it is better not to generate them at all.

10249 07/11/2013 03:48 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: added $is_view

10248 07/11/2013 03:27 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: added $postprocess_sql to store postprocess.sql path, and use it in postprocess()

10246 07/11/2013 02:19 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: import(): allow automated tests (remake_VegBIEN_mappings) to be disabled by setting can_test= if the public schema shouldn't be modified (e.g. if it's the live DB)

10211 07/10/2013 04:13 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: map_table(): run map_table repeatedly until no more renames are made: added command to do this

10210 07/10/2013 03:53 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: map_table(): documented that collisions may prevent all renames from being made at once. if this is the case, map_table must be run repeatedly until no more renames are made. collisions may result if the staging table gets messed up (e.g. due to missing input columns in map.csv).

10171 07/06/2013 02:41 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: need to run remake_VegBIEN_mappings after mk_derived rather than before so the derived cols will be included in the automated test result

10156 07/03/2013 05:56 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: map_table(): store the map table in the datasource schema, so that it can easily be referred to when using the staging tables. this also allows it to be found more easily when debugging its contents.

10151 07/03/2013 05:00 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: map_table(): create the map table as a persistent table in the temp schema, so that its contents can be viewed for debugging

10148 07/03/2013 04:21 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: use new util.set_col_names_with_metadata() instead of util.set_col_names() so that metadata values (beginning with : ) are automatically mapped to constant columns rather than needing to add a mk_const_col() call to postprocess.sql for each of them. there are a lot of metadata value entries, especially in the Source/ tables for each datasource, so this will save time in translating the datasources to new-style import. note that this requires disabling the map_filter_insert trigger on the map table to prevent it from filtering out the metadata entries before util.set_col_names_with_metadata() can use them.

9967 06/20/2013 06:54 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: remake_VegBIEN_mappings(): run yes using piped_cmd() so the SIGPIPE doesn't cause an errexit

9947 06/19/2013 08:24 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: import(): also run remake_VegBIEN_mappings() to accept the test output. this function was previously unused, but was left in for future use when lib/import.sh was translated to lib/runscripts/table.run (it was used in its import.sh form in inputs/FIA/occurrence_all/import).

9946 06/19/2013 08:21 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: remake_VegBIEN_mappings(): need to change to $top_dir before running `rm header.csv map.csv`

9944 06/19/2013 08:00 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: remake_VegBIEN_mappings(): only remake header.csv, map.csv if this target is being run directly, to avoid needing to remake them every time. for tables that are views, this instead requires them to be explicitly remade when the view columns change.

9937 06/19/2013 06:32 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: load_data(): first make sure schema is installed

9936 06/19/2013 06:31 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: added datasrc_make_install()

9935 06/19/2013 06:31 PM Aaron Marcuse-Kubitza

table_make_install(): take $install_log as an overridable kw param to support install logs in different locations

9934 06/19/2013 05:55 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: load_data(): split noclobber functionality into separate table_make_install() function, which can be used by other install-related targets

9854 06/12/2013 03:25 AM Aaron Marcuse-Kubitza

*{.sh,run}: use simpler .rel() instead of `. "$(dirname "${BASH_SOURCE0}")"/...` for relative includes

9847 06/11/2013 07:50 PM Aaron Marcuse-Kubitza

bugfix: load_data(): verbosity_min: use verbosity_min='' so that csv2db's default verbosity (3) is used, instead of setting the verbosity directly to 3, which caused the log++ logging output from bin/make to be echoed at verbosity 3, creating cluttered output

9841 06/11/2013 05:25 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: import(): use self_make on load_data so that the remake status determines whether the table is reinstalled

9839 06/11/2013 05:21 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: load_data(): need to use $_remake instead of $remake when using set_make_vars

9838 06/11/2013 05:20 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: added set_make_vars to all make targets so $remake would be propagated appropriately

9837 06/11/2013 05:18 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: load_data(): also clobber install log if remaking, because the table will be reinstalled

9836 06/11/2013 05:12 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: load_data(): automatically select noclobber mode depending on whether the install log already exists. this removes the need for a separate load_data_first_run() function.

9835 06/11/2013 04:59 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: load_data(): ignore errors if table already exists

9834 06/11/2013 04:52 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: load_data(): use noclobber=1 to avoid overwriting the install log when re-running the install target idempotently. load_data_first_run() is now available to preserve the output in the log on the first run.

9681 06/01/2013 05:17 AM Aaron Marcuse-Kubitza

bugfix: *run: overriding targets: use new self_make to properly progagate the $remake flag to the overridden target, so that the target itself is not skipped

9625 05/29/2013 08:02 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run, table.run: use new db_make.sh

9585 05/24/2013 01:43 PM Aaron Marcuse-Kubitza

lib/runscripts/table_dir.run: renamed table to subdir because this can apply to any datasrc subdir. moved table-specific code to table.run.

9555 05/24/2013 01:22 AM Aaron Marcuse-Kubitza

added lib/runscripts/table_dir.run and use it in table.run

9469 05/20/2013 07:27 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: include make.sh so runscripts based on it can use make-related utils

9428 05/16/2013 07:01 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: don't mk_esc_name schema, table because these will be mk_esc_name'd by functions that use them

9408 05/16/2013 03:02 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: load_data(): use new $verbosity_min instead of running `verbosity_min` so that the command name logging is not output with the new verbosity

9398 05/16/2013 06:15 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: load_data(): need to ensure the verbosity is at least 3 because the install logs require verbose output. (3 is the default for the installer, but is overridden by the runscripts, which instead set the default to 1.)

9394 05/16/2013 01:59 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: $schema, $table: log the `cd` used to calculate the value at log_level 3 instead of 1 (note that the cd() function call for this will be logged at log_level 5)

9392 05/15/2013 11:28 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: import(): added step to load the data into the staging table before postprocessing it

9388 05/15/2013 07:51 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: table_make(): run canon_rel_path on the datasrc dir so it's displayed as the direct path to it, without ..

9387 05/15/2013 07:47 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: table_make(): moved comment about "${@/#/$table/}" to right after the line it describes

9386 05/15/2013 07:44 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): renamed to table_make() to make it clear that the target names are relative to the table subdir itself, not the datasrc dir. it was previously called input_make because it used inputs/input.Makefile directly, but now will use any Makefile in the datasrc dir.

9355 05/15/2013 09:27 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): added $silent flag which turns on make's --silent option

9352 05/15/2013 08:46 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): documented that it requires a Makefile in the datasrc dir containing `include ../input.Makefile

9351 05/15/2013 08:41 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): added comment explaining that "${@/#/$table/}" replaces the empty str at the beginning of str (/#) with $table/

9350 05/15/2013 08:38 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: put functions on one line where possible

9349 05/15/2013 08:35 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: input_make(): use the local make() function instead of external make directly, because it sets $cmd_log_fd appropriately to ensure that all the echoed make commands get properly logged to stdlog/stderr (stdlog is fd 30 when it's redirected to a file)

9169 05/11/2013 01:36 PM Aaron Marcuse-Kubitza

*{.sh,run}: use command instead of deprecated echo_run (don't prepend anything when the command is already aliased with `command`)

9075 05/10/2013 12:14 PM Aaron Marcuse-Kubitza

*{.sh,run}: replaced extern with command, a shell builtin that does the same thing. this is also part of dash (the Bourne shell on Linux).

9074 05/10/2013 12:08 PM Aaron Marcuse-Kubitza

*{.sh,run}: removed extra space between function name and (), which is apparently not needed even though `help function` includes it. this greatly improves readability, including when function names are pasted into commit messages!

9047 05/10/2013 08:41 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: remake_VegBIEN_mappings (): added public_schema_exists check, ported from lib/import.sh

9039 05/10/2013 07:39 AM Aaron Marcuse-Kubitza

bugfix: : use new func_override for runscript inheritance instead of invoking the overridden function as a (command-line) target of the parent runscript. this ensures that the overridden function is run in the *same process as the calling function, so that $top_dir keeps the same value and runscript-relative paths continue to work.

9013 05/10/2013 04:05 AM Aaron Marcuse-Kubitza

moved lib/*.sh to sh/ subdir so it's easier to find the .sh files among all the other lib/ files

8910 05/09/2013 05:57 AM Aaron Marcuse-Kubitza

lib/util.sh: use extern instead of env to run external commands, because it's simpler and auto-captures the logging output

8896 05/09/2013 03:05 AM Aaron Marcuse-Kubitza

renamed lib/runscripts/local.run to lib/local.sh since the things it defines are not just for runscripts

8881 05/08/2013 08:02 PM Aaron Marcuse-Kubitza

*{.sh,run}: changed echo_func to an alias that includes the "$", so that callers don't need to specify the "$" manually. although the original echo_func function is still there, callers need to switch over to the alias at the same time as it's defined because otherwise the "$@" would be doubled, since echo_func refers to the alias instead.

8873 05/08/2013 06:02 PM Aaron Marcuse-Kubitza

*{.sh,run}: use just env instead of echo_run env now that env is an auto-echoing alias

8775 05/02/2013 05:13 AM Aaron Marcuse-Kubitza

lib/runscripts/table.run: moved general DB commands to lib/util.sh

8774 05/02/2013 04:47 AM Aaron Marcuse-Kubitza

moved lib/*.run into runscripts/ subdir so runscripts are grouped together and easier to find rather than being scattered throughout lib/

8705 04/25/2013 04:49 PM Aaron Marcuse-Kubitza

*run: added include guards

8701 04/25/2013 03:37 PM Aaron Marcuse-Kubitza

lib/table.run: use new local.run instead of defining $bin_dir, etc. itself

8698 04/25/2013 02:50 PM Aaron Marcuse-Kubitza

lib/table.run: added mysql (), mysql_ANSI ()

8696 04/25/2013 04:54 AM Aaron Marcuse-Kubitza

lib/table.run: run mk_esc_name for $schema, $table

8695 04/24/2013 11:18 PM Aaron Marcuse-Kubitza

lib/table.run: added esc_name (), mk_esc_name ()

8463 04/13/2013 02:28 AM Aaron Marcuse-Kubitza

lib/util.run: echo_func (): use the caller's FUNCNAME via the $FUNCNAME[] array instead of requiring them to pass it in the function args as `echo_func "$FUNCNAME" "$@"`