Project

General

Profile

Statistics
| Revision:
  • svn:ignore: *.pyc tnrs.url

# Date Author Comment
13597 06/02/2014 04:24 PM Aaron Marcuse-Kubitza

fix: lib/tnrs.py: Constrain by Source: turn it on so that the download settings reflect what TNRS actually used, while this is broken

13596 06/02/2014 06:19 AM Aaron Marcuse-Kubitza

fix: lib/tnrs.py: max_names: reduced back to 500 because even 5000 crashes the dev TNRS server

13595 06/02/2014 05:52 AM Aaron Marcuse-Kubitza

lib/tnrs.py: max_names: reduced to 5000 because 100,000 causes an internal server error

13591 06/02/2014 04:50 AM Aaron Marcuse-Kubitza

lib/tnrs.py: switched to downloading all matches per name, as is needed to implement #917. note that this will break the parts of the schema that use the tnrs table, until Brad's match-picking algorithm can be implemented, but this tradeoff is necessary to be able to begin scrubbing sooner (Martha; wiki.vegpath.org/2014-05-29_conference_call#TNRS)

13562 05/30/2014 07:50 AM Aaron Marcuse-Kubitza

lib/tnrs.py: max_names: increased to 100000 because the dev server can handle more names (no simultaneous users), as decided in the conference call (wiki.vegpath.org/2014-05-29_conference_call#TNRS)

13559 05/30/2014 07:37 AM Aaron Marcuse-Kubitza

fix: lib/PostgreSQL-MySQL.csv: need to replace "double precision" with "double" to work with MySQL Workbench 5.2.47

13548 05/29/2014 11:53 AM Aaron Marcuse-Kubitza

lib/tnrs.py: commented out the value of max_names that is not active, for clarity

13544 05/27/2014 11:12 PM Aaron Marcuse-Kubitza

lib/tnrs.py: sources: updated to list/sort order in issue #917

13500 05/21/2014 01:23 AM Aaron Marcuse-Kubitza

fix: lib/PostgreSQL-MySQL.csv: also remove left-behind lines such as `$$);`

13473 05/17/2014 06:22 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: $is_runscript: unexport so don't pass it to invoked scripts

13465 05/17/2014 02:15 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: run_args_cmd(): don't prepend main to args if no args, because for a non-runscript, all args will be passed to main(), leading `main` to be doubled

13464 05/17/2014 01:30 PM Aaron Marcuse-Kubitza

lib/tnrs.py: use the TNRS dev server (with private URL in tnrs.url) instead of the live server, since that contains datasources that we need

13463 05/17/2014 01:29 PM Aaron Marcuse-Kubitza

lib/streams.py: added file_get_contents()

13462 05/17/2014 01:14 PM Aaron Marcuse-Kubitza

lib/tnrs.py: configure the server separately from the base URL

13461 05/17/2014 01:12 PM Aaron Marcuse-Kubitza

lib/: svn:ignore tnrs.url so the TNRS dev server URL does not become public

13436 05/12/2014 07:06 PM Aaron Marcuse-Kubitza

lib/tnrs.py: retrieval_request_template: taxonomic_constraint, source_sorting: documented their meaning and why they need to be on/off

13413 05/07/2014 05:18 PM Aaron Marcuse-Kubitza

lib/runscripts/import.run: added install() target

13409 05/07/2014 03:29 PM Aaron Marcuse-Kubitza

lib/runscripts/in_datasrc_dir.run: use new local.run

13408 05/07/2014 03:25 PM Aaron Marcuse-Kubitza

added lib/runscripts/local.run

13392 05/02/2014 10:57 PM Aaron Marcuse-Kubitza

fix: lib/util.py: dict_subset(): raise an error if collections.OrderedDict isn't available, because some callers may depend on this. note that using dict instead of OrderedDict may be the cause of the joining on the wrong columns bug (issue #902).

13373 05/01/2014 01:37 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/validations.pg.sql.run: updated to reflect that validations.sql is now located inside a subdir, not the datasrc dir

13372 05/01/2014 01:29 PM Aaron Marcuse-Kubitza

fix: lib/runscripts/file.pg.sql.run: removed include of in_datasrc_dir.run, because this location does not apply to all .sql export scripts

13367 05/01/2014 04:09 AM Aaron Marcuse-Kubitza

lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability

13365 05/01/2014 03:14 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: pg_dump(): need use_pg to import $pg_database before checking for existence of $database

13364 05/01/2014 03:11 AM Aaron Marcuse-Kubitza

lib/sh/util.sh: import_vars: documented that it's idempotent

13361 04/30/2014 06:58 PM Aaron Marcuse-Kubitza

bugfix: lib/util.py: use OrderedDict from collections rather than ordereddict to work with Mac OS X 10.8 Mountain Lion (http://vegpath.org/links/#OrderedDict)

13354 04/29/2014 11:36 PM Aaron Marcuse-Kubitza

bugfix: benign_does_not_exist_error(): removed ignore_e=3, because this exit status is also used for other errors

13353 04/29/2014 11:35 PM Aaron Marcuse-Kubitza

fix: lib/sh/db.sh: benign_does_not_exist_error(): use benign_error=1, which is now supported properly by stderr_matches()

13352 04/29/2014 11:34 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: stderr_matches(): support $benign_error properly, by handling exit status logging in this func instead

13351 04/29/2014 11:03 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: pg_schema_exists(): also need to benignify "does not exist" error if returns false

13350 04/29/2014 10:42 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: stderr_matches(): need to separately display errors that were incorrectly suppressed due to $benign_error

13349 04/29/2014 10:36 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: is_err(): rethrow must be inverted (rethrow->false if error)

13348 04/29/2014 10:32 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added is_err()

13347 04/29/2014 09:53 PM Aaron Marcuse-Kubitza

lib/sh/local.sh: public_schema_exists(): moved to lib/sh/db.sh since this no longer depends on BIEN-specific configurations

13346 04/29/2014 09:42 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: public_schema_exists(): don't hide the function call tree so it's clear which function is running the psql commands

13345 04/29/2014 09:40 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: public_schema_exists(): don't hide the function call tree so it's clear which function is running the psql commands

13344 04/29/2014 09:38 PM Aaron Marcuse-Kubitza

fix: *{.sh,run}: stderr_matches() callers: added benign_error=1 where needed

13343 04/29/2014 09:36 PM Aaron Marcuse-Kubitza

fix: *{.sh,run}: stderr_matches() callers: usage: documented that they may require benign_error=1

13342 04/29/2014 09:35 PM Aaron Marcuse-Kubitza

fix: lib/sh/util.sh: stderr_matches(): usage: documented that this may require benign_error=1

13340 04/29/2014 09:23 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: pg_cmd(): updated for new echo_vars log_level

13339 04/29/2014 09:14 PM Aaron Marcuse-Kubitza

fix: lib/sh/db.sh: pg_schema_exists(): display the function name so it's clear which function is running the psql commands

13338 04/29/2014 05:37 PM Aaron Marcuse-Kubitza

fix: lib/sh/db.sh: pg_schema_exists(): don't use log++ because it hides the command that produces the benign error

13328 04/25/2014 11:38 PM Aaron Marcuse-Kubitza

fix: lib/sh/db.sh: psql(): removed debugging changes

13327 04/25/2014 11:36 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: highlight_log_msg(): when not can_highlight_log_msg, need to remove any surrounding formatting

13326 04/25/2014 11:01 PM Aaron Marcuse-Kubitza

fix: lib/sh/util.sh: die_error_hidden(): always log local vars at same log_level as echo_func

13325 04/25/2014 10:56 PM Aaron Marcuse-Kubitza

fix: *{.sh,run}: always log kw_params at same log_level as echo_func

13324 04/25/2014 06:27 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: split_lines(): usage: matched up and synced different syntaxes

13323 04/25/2014 06:22 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: log_msg!(): split_lines does not support being invoked by wrapper; need to use `declare lines; wrapper "split_lines" str` instead

13322 04/25/2014 06:21 PM Aaron Marcuse-Kubitza

fix: lib/sh/util.sh: split_lines(): usage: documented different syntax for when using wrapper

13321 04/25/2014 06:09 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: die_error_hidden(): echo_func to assist debugging

13320 04/25/2014 06:07 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: split(): need to limit the effects of IFS to just the splitting, so it doesn't cause strange errors in other functions

13319 04/25/2014 03:20 PM Aaron Marcuse-Kubitza

bugfix: stderr2stdout(): fd 2 must be redirected back to fd 2, not log-filtered, in case there are other errors in addition to the benign error. this fixes a bug in pg_schema_exists(), where errors about the DB being down were not displayed because they were log-filtered out.

13318 04/25/2014 06:15 AM Aaron Marcuse-Kubitza

lib/sh/make.sh: set_make_vars: don't display make vars at verbosity 2 to avoid clutter

13317 04/24/2014 05:34 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/make.sh: begin_target: don't echo_func twice

13313 04/24/2014 03:49 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: a non-runscript should have all args passed to main(). this fixes a bug in backups/*_snapshot where "main" would need to be prepended to any args for the script to run correctly.

13312 04/24/2014 03:43 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: $wrap_fn: invoked script must always run as runscript so that wrapped command is run

13311 04/24/2014 03:40 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: added $is_runscript, for use by $wrap_fn

13310 04/24/2014 03:36 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/util.run: $wrap_fn: $top_script doesn't need to be world-executable for most uses of sudo (only if sudoing to non-root)

13288 04/22/2014 10:33 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: pg_snapshot(): perform online backup if possible, to avoid needing to restart the server

13287 04/22/2014 10:31 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: added pg_start_backup(), pg_stop_backup()

13286 04/22/2014 10:29 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: psql(): only set redirs if can redirect

13285 04/22/2014 10:17 PM Aaron Marcuse-Kubitza

bugfix: psql(): when as_root is on, need to avoid redirections as these are not passed through by sudo

13283 04/22/2014 06:01 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: psql(): don't use `--file /dev/fd/40` when can't redirect

13282 04/22/2014 05:59 PM Aaron Marcuse-Kubitza

fix: lib/sh/db.sh: psql(): when using sudo with custom $stdin, raise error that this is not supported

13281 04/22/2014 05:50 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: psql(): $data_bypasses_filter: renamed to $data2stdout for clarity

13280 04/22/2014 05:29 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: psql(): $bypass_ok: renamed to $can_redir for clarity

13279 04/22/2014 05:22 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: psql(): usage: $stdin: documented that can also use process substitution for this

13278 04/22/2014 04:45 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: `type` calls: need -- before cmd in case it starts with -

13277 04/22/2014 04:44 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: cmd2rel_path: use is_extern()

13276 04/22/2014 04:00 PM Aaron Marcuse-Kubitza

lib/sh/sync.sh: db_snapshot(): use `end_try` as specified in `try` usage

13275 04/22/2014 03:59 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: try: usage: added location of finally block

13273 04/22/2014 03:04 PM Aaron Marcuse-Kubitza

lib/sh/sync.sh: upload(): always print the function and kw_params

13271 04/21/2014 08:21 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: type(): need to handle options before command name

13270 04/21/2014 08:21 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added 1st_non_opt()

13269 04/21/2014 08:07 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: unalias(): use self_builtin, which is now defined before it

13268 04/21/2014 08:01 PM Aaron Marcuse-Kubitza

lib/runscripts/util.run: sudo(): avoid slow $wrap_fn when using `command` (ie. always executable)

13267 04/21/2014 07:59 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: unalias(): use self_builtin, which is now defined before it

13266 04/21/2014 07:57 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: commands: `type` calls: need to account for the fact that any alias is already expanded

13265 04/21/2014 07:56 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: functions: moved before commands since commands are more complex

13264 04/21/2014 07:38 PM Aaron Marcuse-Kubitza

lib/sh/sync.sh: db_copy() and callers: pass args as rsync options

13263 04/21/2014 07:28 PM Aaron Marcuse-Kubitza

fix: lib/sh/sync.sh: db_copy(): need to exclude files which prevent tape backup

13262 04/21/2014 07:22 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: added pg_ctl(), pg_snapshot()

13261 04/21/2014 07:17 PM Aaron Marcuse-Kubitza

lib/sh/sync.sh: db_snapshot(): copy changes before stopping DB to minimize the time that it's shut down

13260 04/21/2014 07:12 PM Aaron Marcuse-Kubitza

lib/sh/sync.sh: db_snapshot(): factored copy operation out into separate db_copy() function

13259 04/21/2014 07:03 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql_snapshot(): use new db_snapshot()

13258 04/21/2014 07:02 PM Aaron Marcuse-Kubitza

lib/sh/sync.sh: added db_snapshot()

13257 04/21/2014 05:55 PM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: changed "page's own description" to "page's self-description" for clarity

13253 04/21/2014 04:46 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql_snapshot(): for large files, don't re-copy entire file

13252 04/21/2014 04:44 PM Aaron Marcuse-Kubitza

lib/sh/db.sh: mysql_snapshot(): use live mode as the default

13251 04/21/2014 04:30 PM Aaron Marcuse-Kubitza

fix: lib/sh/db.sh: mysql_snapshot(): need to create dest dir if doesn't exist

13250 04/21/2014 04:27 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: mysql_snapshot(): try: need to use split syntax with prep_try instead, to work with prefix vars

13249 04/21/2014 04:23 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/db.sh: mysql_snapshot(): try: need to use split syntax with prep_try instead, to work with prefix vars

13248 04/21/2014 04:20 PM Aaron Marcuse-Kubitza

fix: lib/sh/util.sh: try usage: documented that the split syntax with prep_try is meant to be used with vars before the cmd

13247 04/21/2014 03:37 PM Aaron Marcuse-Kubitza

fix: lib/sh/util.sh: echo_vars(): also need to print unset vars (including unset kw_params)

13246 04/21/2014 03:31 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_vars(): put loop var on same line as `for`

13245 04/21/2014 02:59 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: sudo(): need to preserve PATH separately because -E does not preserve this

13244 04/21/2014 02:17 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_redirs_cmd(): inline the function alias since it's only used in one place

13243 04/21/2014 02:15 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: redir(): need to load new aliases before it

13242 04/21/2014 02:13 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_redirs_cmd(): log $PATH to facilitate troubleshooting

13241 04/21/2014 01:54 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: echo_redirs_cmd(): documented what the $(...) section does