Project

General

Profile

Statistics
| Revision:
  • svn:ignore: *.pyc

# Date Author Comment
11970 01/20/2014 11:33 AM Aaron Marcuse-Kubitza

moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).

11914 12/16/2013 04:58 PM Aaron Marcuse-Kubitza

fix: lib/sh/util.sh: verbosity_min(): usage: clarified that '' is a special value that causes $verbosity to be overwritten to ''

11913 12/16/2013 04:45 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: added test_() target and use it in remake_VegBIEN_mappings() (it would not be clear that remake_VegBIEN_mappings() runs the tests)

11895 12/11/2013 07:43 PM Aaron Marcuse-Kubitza

lib/common.Makefile: added %/live, for use with `make inputs/download`

11875 12/09/2013 05:44 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/datasrc_dir.run: import(): don't run `sql/install` if the schema already exists, because this will try to rerun all the schema-creation queries. note that this idempotent functionality was not provided by the `make .../install` target that was previously used (idempotency is new with new-style import).

11872 12/09/2013 03:54 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/datasrc_dir.run: import(): can't run `datasrc_make reinstall` anymore because this now defers to the runscript for new-style import datasources (which was done so that `make .../install` properly reinstalls all the datasources). instead, call the applicable make targets manually (there are just 2 of them).

11861 12/06/2013 05:52 AM Aaron Marcuse-Kubitza

lib/sh/local.sh: public_schema_exists(): use a higher log_level for pg_schema_exists, to avoid all the verbose output involved in running the query

11860 12/06/2013 05:44 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: public_schema_exists(): can no longer use psql_script_vegbien for this, because using `SET search_path` (called by psql_script_vegbien) with a schema that does not exist no longer produces an error. instead, use new pg_schema_exists(), which uses a different command that does produce an error if the schema does not exist.

11859 12/06/2013 05:38 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: added pg_require_schema()

11858 12/06/2013 05:37 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: stderr2stdout(): documented that this redirects fd 2->1 and log_fd (but not back to 2)

11857 12/06/2013 05:34 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: stderr2stdout() use `command` before tee, which re-filters log_fd so that stderr itself is also filtered. this allows log-filtering out an otherwise-confusing benign error when using e.g. stderr_matches().

11856 12/06/2013 04:31 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added not(), for use in prefixing wrapped commands

11855 12/06/2013 04:14 AM Aaron Marcuse-Kubitza

lib/sh/db.sh: added pg_schema_exists()

11854 12/06/2013 04:10 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added stderr_matches()

11853 12/06/2013 03:59 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: documented that fds 2x/3x should not be used because we use these, as opposed to 1x which is used by the shell internally

11852 12/06/2013 03:57 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added stdout_contains()

11851 12/06/2013 03:34 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: added stderr2stdout()

11850 12/06/2013 02:52 AM Aaron Marcuse-Kubitza

fix: lib/sh/db.sh: pg_table_exists(): usage: documented that $table is actually required for this function

11784 11/26/2013 10:58 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: import_vars: don't overwrite vars that are already defined, to allow the caller to specify their own values for the vars to create. this requires callers that rely on the overwriting functionality to reverse the order in which they run use_* commands, so that the higher-precedence use_* is applied first and the other one as the default values for the first.

11771 11/26/2013 01:22 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: pg_table_exists(): use `SELECT NULL` instead of `SELECT *` to avoid a long column list cluttering up the log output

11770 11/26/2013 12:47 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: table_make_install(): simplified the setting of $noclobber since there no longer needs to be a different command for when the log exists

11769 11/26/2013 12:08 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: need to errexit the make target, so that errors in the SQL install scripts are not suppressed. this requires pre-checking if the table exists (using new pg_table_exists), so that the install target's errexit does not then need to be suppressed for cases when the table already exists.

11768 11/26/2013 12:01 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: added pg_table_exists()

11665 11/14/2013 07:19 AM Aaron Marcuse-Kubitza

bugfix: lib/Firefox_bookmarks.reformat.csv: remove empty <DD> tags (which Firefox now adds for all bookmarks) so they don't create a blank space on the page

11664 11/14/2013 07:16 AM Aaron Marcuse-Kubitza

bugfix: lib/Firefox_bookmarks.reformat.csv: don't prepend "page's description:" to empty <DD> tags, which Firefox now adds for all bookmarks, even if they don't have a description

11662 11/14/2013 06:44 AM Aaron Marcuse-Kubitza

Makefile, schemas/.Mac.conf: upgraded to PostgreSQL 9.3, which is needed for proper exception parsing in the auto-re-create-views functionality. this also removes the Mac 10.8 Mountain Lion quirks, such as renaming the postgres user to _postgres (which messed everything up, but is now back to normal).

11624 11/10/2013 12:04 AM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/file.pg.sql.run: export_(): exclude Source and related tables so that these will be re-created by the staging tables installation instead, ensuring that they are always in sync with the Source/ subdir

11605 11/09/2013 02:02 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: already_exists_msg(): added instructions on how to force-remake when the file already exists (prepend `rm=1` to the command)

11603 11/08/2013 10:57 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/view.run: don't do anything in load_data(), to avoid trying to remake header.csv before the view is created. (for views, this instead happens in postprocess().)

11602 11/08/2013 10:51 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: reordered functions in the order they are called by import()

11588 11/06/2013 02:39 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/make.sh: $remake: need to explicitly propagate this to invoked commands if it was set from $rm

11579 11/05/2013 11:13 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mk_select(): usage: documented that this also takes a $limit/$n param

11578 11/05/2013 11:12 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: limit(): also support using $n as the limit param, since this var name is used by other parts of the import process

11576 11/05/2013 11:07 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: limit(): usage: documented that this also need a $limit param

11574 11/05/2013 10:47 PM Aaron Marcuse-Kubitza

lib/runscripts/extract.run: added export_sample()

11462 10/29/2013 07:46 AM Aaron Marcuse-Kubitza

lib/runscripts/import_subset.run: $version: use new $extract_view, which is set to the same value that this was

11461 10/29/2013 07:45 AM Aaron Marcuse-Kubitza

lib/runscripts/extract.run: use the extract-specific view instead of all of analytical_stem

11436 10/24/2013 07:08 PM Aaron Marcuse-Kubitza

added lib/runscripts/import_subset.run, extract.run

11433 10/24/2013 05:06 PM Aaron Marcuse-Kubitza

lib/sh/local.sh: psql(): also accept $public as the $schema param, since this is used by a lot of import scripts

11432 10/24/2013 04:24 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added require_dot_script()

11431 10/24/2013 04:13 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: $top_script: use @BASH_SOURCE instead of $0, because this is also defined for .-scripts

11428 10/24/2013 03:26 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: `trap on_exit EXIT`: only set this if the script is not a dot script, because if it is a dot script, on_exit() will not be invoked until the calling shell exits, which may be much later than when the script is run. previously, this was handled by canceling the EXIT trap if on_exit() is run manually, but this would not work correctly if a load-time error prevented on_exit() from running and canceling the trap.

11427 10/24/2013 03:21 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: if is_dot_script, fix $ when no args causes this to incorrectly contain the script name. use is_dot_script rather than the presence of $ args to decide whether to use @BASH_ARGV, because @BASH_ARGV is actually wrong when run as a .-script (it contains the script name).

11426 10/24/2013 03:17 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: is_dot_script(): need to subtract 1 from ${#BASH_LINENO[@]}, because this is the array length rather than the index of the last element as in Perl

11425 10/24/2013 02:58 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added is_dot_script()

11418 10/23/2013 11:02 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: run script template: changed sample command name to all() because each runscript requires this in order to be run without args

11417 10/23/2013 11:00 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by allowing the calling script to manually invoke on_exit() without it then being invoked twice (the end of a shell-include does not trigger the EXIT trap)

11414 10/23/2013 10:17 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: support scripts that are run as shell-includes (with leading "."), by also accepting $@ args that are passed along in the util.run include, in addition to @BASH_ARGV

11413 10/23/2013 09:11 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: alias_append(): need to enclose $(alias) call in "" because its result may contain separator chars (i.e. whitespace) that will be parsed incorrectly. this appears to only be a bug when runscripts are run as shell-includes, with a leading ".".

11365 10/19/2013 12:52 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: psql(): $is_root: use `` around case statement instead of $(), because it contains an embedded unbalanced )

11362 10/18/2013 10:32 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: psql(): don't default the connection vars using use_local if running as the postgres user. in that case, connection must happen via a socket, with server="", and as the user running the command (postgres), with user="".

11360 10/18/2013 09:54 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: avoid outputting to /dev/fd/# when running as sudo on Linux, because this causes a "Permission denied" error (due to the /dev/fd/# file being owned by a different user). this is not a problem with normal redirects (>&#), because they do not use /dev/fd/# files which can have access permissions.

11359 10/18/2013 09:52 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: to_top_file(): need to pass "$@" to to_file

11358 10/18/2013 08:17 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: to_top_file: added function for this (in addition to alias), so that this can be run from sudo in a wrap_fn command

11357 10/18/2013 07:50 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: pg_as_root(): run sudo with echo_run to help debug

11356 10/18/2013 06:29 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: pg_cmd(): only set PG* connection/login env vars when the corresponding var is non-empty. there are some situations in which these must be unset (in order to use the default value), and other situations when the var must be set to something (i.e. "") to avoid it being defaulted to a value in local.sh > connection vars.

11353 10/18/2013 06:11 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: pg_as_root(): need to use -E (preserve environment) option to sudo, so that $schema, $table get passed through

11352 10/18/2013 06:05 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/local.sh: psql(): only \set schema, table if $schema, $table are non-empty, because otherwise, you will get a "zero-length delimited identifier" error

11350 10/18/2013 05:29 PM Aaron Marcuse-Kubitza

lib/sh/local.sh: added require_remote()

11349 10/18/2013 05:29 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: added pg_as_root()

11348 10/18/2013 05:28 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: added $wrap_fn to run any function via sudo, etc.

11266 10/13/2013 12:46 PM Aaron Marcuse-Kubitza

bugfix: lib/common.Makefile: $(subMake): don't enclose the target in "" because sometimes the target is empty (i.e. `all`), and nothing should be passed to the sub-make

11253 10/12/2013 12:48 PM Aaron Marcuse-Kubitza

bugfix: *Makefile: recursive invocation of $(MAKE): enclose targets in "" in case they contain *

11252 10/12/2013 12:12 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: load_data(): pass $is_view through to `make reinstall` so that DROP VIEW will be used instead of DROP TABLE where applicable

11250 10/12/2013 12:08 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added instructions for making an export only visible locally

11247 10/10/2013 05:46 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/table.run: load_data(): in remaking mode, need to remake header.csv in case the columns have changed

11170 10/08/2013 05:31 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: import(): added remake (rm=1) mode that reinstalls the datasource before continuing with the subdirs' import actions

11169 10/08/2013 05:29 PM Aaron Marcuse-Kubitza

lib/runscripts/in_datasrc_dir.run: added datasrc_make(), which runs make in the datasrc dir

11151 10/02/2013 02:45 AM Aaron Marcuse-Kubitza

lib/sql_io.py: put_table(): default param: documented that this will be used for all missing rows, regardless of which error caused them not to be inserted. this means that auto-forwarding (wiki.vegpath.org/Auto-forwarding) can be used with any type of constraint violation, not just NOT NULL constraints (which it is typically used with).

11033 09/21/2013 09:01 PM Aaron Marcuse-Kubitza

lib/sql_io.py: put_table(): added link to new INSERT ON DUPLICATE SELECT wiki page, which now contains the explanation in the doc comment

11000 09/17/2013 10:05 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/*: calls to rm: use `rm -f` instead to avoid an error (which aborts the program) if the file does not yet exist

10996 09/16/2013 12:13 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/make.sh: don't allow rm to override remake if an invoked script uses this file. this fixes a bug in `rm=1 inputs/.../.../run` where the remake action would be invoked on the map_table command even though it had been suppressed, because it was run externally (i.e. make.sh was reloaded) and the rm=1 flag was still active

10946 09/14/2013 04:20 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: added postprocess target to run postprocess in just the table subdirs, skipping any additional subdirs that don't have this target

10945 09/14/2013 04:19 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run: @subdirs: moved import_order.txt subdirs into separate @table_subdirs, which provides access to just the table subdirs when the user adds other dirs to @subdirs

10929 09/12/2013 03:34 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/view.run: remake_VegBIEN_mappings(): also need to remake header.csv, not just map.csv as for tables, because view columns may change when the view is regenerated

10921 09/12/2013 12:35 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: set_fds(): don't add surrounding quotes to empty redirect dest

10920 09/12/2013 12:31 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: set_fds(): need to check if redirect is empty before escaping it with `printf %q`, which may add surrounding quotes to an empty string

10890 09/07/2013 08:14 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: set_fds(): need to escape redirect destinations which are files, because they may contain special shell characters

10889 09/07/2013 08:10 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added rm_prefix()

10888 09/07/2013 07:11 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql_cmd(): added caller usage with connection/login opts

10887 09/07/2013 07:08 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql(), mysql_export(): usage: added database=...

10882 09/05/2013 03:56 PM Aaron Marcuse-Kubitza

bugfix: schemas/Makefile, lib/common.Makefile: enclose schema names in "" so that they won't be lowercased

10845 08/31/2013 06:32 PM Aaron Marcuse-Kubitza

bugfix: lib/sql_io.py: put_table(): Getting output table pkeys of existing/inserted rows: need to include the index cond in the join condition here, too (using var join_custom_cond), so that an index scan can be used instead of a much slower full-table sort

10843 08/31/2013 05:52 PM Aaron Marcuse-Kubitza

bugfix: lib/sql_io.py: put_table(): DuplicateKeyException: need to include any index cond in the join condition, so that an index scan can be used instead of a much slower full-table sort (otherwise the query planner will not know that it can restrict results to rows satisfying the index cond)

10842 08/31/2013 05:48 PM Aaron Marcuse-Kubitza

lib/sql_gen.py: Join: added custom_cond param that can be used to add to the JOIN condition

10841 08/31/2013 01:02 AM Aaron Marcuse-Kubitza

lib/sql.py: distinct_table(): support custom filters on the distincting query

10840 08/31/2013 01:01 AM Aaron Marcuse-Kubitza

lib/sql_gen.py: ColValueCond: support conds that are just plain SQL (without separate left and right sides) using special custom_cond flag value

10838 08/30/2013 10:38 PM Aaron Marcuse-Kubitza

lib/sql_io.py: ensure_cond(): documented meaning of passed, failed params (at least one row passed/failed the constraint)

10835 08/30/2013 06:06 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: usage: documented that this usage also applies to all files that include this file

10834 08/30/2013 06:06 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: usage: clarified that the cmd to run is a function

10832 08/30/2013 06:02 PM Aaron Marcuse-Kubitza

added lib/runscripts/pg.conf.run, which installs PostgreSQL config files

10831 08/30/2013 06:01 PM Aaron Marcuse-Kubitza

added lib/runscripts/install.run, analogous to import.run

10829 08/30/2013 04:35 PM Aaron Marcuse-Kubitza

bugfix: lib/db_xml.py: put_table(): turned off db.autoanalyze, since forcing an ANALYZE after every bulk insert is inefficient for small datasources. the default autovacuum settings in schemas/postgresql.conf should be fine; however, the frequency and/or threshold may need to be increased if autovacuum does not ANALYZE frequently enough to replace db.autoanalyze.

10827 08/30/2013 03:36 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mk_select(): added support for ORDER BY

10825 08/30/2013 03:28 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: added pg_export_table_to_dir(), analogous to pg_export_table_to_dir_no_header()

10822 08/30/2013 02:18 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: is_array(): handle unset vars (=false). this fixes a bug in pg_export_table_no_header, which produced the error "lib/sh/util.sh: line 290: declare: cols: not found".

10821 08/30/2013 02:06 PM Aaron Marcuse-Kubitza

fix: lib/sh/util.sh: join(): documented that delim must be a single char

10781 08/27/2013 09:35 PM Aaron Marcuse-Kubitza

added lib/runscripts/data.pg.sql.run (analogous to schema.pg.sql.run for data-only SQL scripts)

10780 08/27/2013 09:32 PM Aaron Marcuse-Kubitza

added lib/runscripts/file.pg.sql.run and use it in schema.pg.sql.run

10779 08/27/2013 09:25 PM Aaron Marcuse-Kubitza

added lib/runscripts/schema.pg.sql.run and use it in inputs/.TNRS/schema.sql.run