Activity
From 04/13/2014 to 05/12/2014
05/12/2014
- 07:06 PM Revision 13436: lib/tnrs.py: retrieval_request_template: taxonomic_constraint, source_sorting: documented their meaning and why they need to be on/off
- 05:03 PM Task #916: Taxon name validation: VegBank
- regarding:
Line 617: TNRS gives a synonym for Aronia prunifolia in a different genus, but this is missed here
L... - 04:35 PM Task #916 (New): Taxon name validation: VegBank
- Hi Aaron,
Bob only had time to get part way through the VegBank taxon validation file you sent, but there are some... - 04:28 PM Revision 13435: schemas/public_.sql: _plots_20_tnrs_names: changed to set of columns requested by Martha
- 03:13 PM Task #915 (New): Taxon Name Validation
- 1. Please correct the following in taxonMorphospecies:
1.1. -Family should be included in taxonMorphospecies only ...
05/10/2014
- 09:32 PM Revision 13434: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB
- 09:17 PM Revision 13433: schemas/public_.sql: _plots_20_tnrs_names: added morphospecies suffix in order to also validate it
- 09:13 PM Revision 13432: schemas/public_.sql: _plots_20_tnrs_names: added verbatim name in order to also validate the formation of the concatenated name
- 08:49 PM Revision 13431: schemas/public_.sql: merged _plots_20_tnrs_input_names, _plots_21_tnrs_output_names into _plots_20_tnrs_names so the input and output names can be compared side-by-side
- 08:10 PM Revision 13430: schemas/vegbien.sql: synced to DB, which adds Source.errors table indexes added in the import
05/09/2014
- 07:27 PM Revision 13429: /README.TXT: Full database import: for test import, turn off DB backup: added () after after_import to show that it's a function
- 07:24 PM Revision 13428: /README.TXT: Full database import: refer to the shell instead of screen because it's also possible to use a nested shell for this
- 02:51 AM Revision 13427: bugfix: /README.TXT: Full database import: to import just a subset of the datasources: need to set @inputs separately from declaring it, because declare does not support () on Mac
- 02:16 AM Revision 13426: fix: /README.TXT: Full database import: on local machine: need to allow logging output to go to log files
- 02:05 AM Revision 13425: /README.TXT: Full database import: for test import, added step to turn off DB backup
- 01:48 AM Revision 13424: fix: /README.TXT: Full database import: setting $version: moved to inside screen to avoid affecting the outer shell
- 01:46 AM Revision 13423: /README.TXT: Full database import: screen: added alternative for a small import (just use nested shell)
- 01:42 AM Revision 13422: fix: /README.TXT: Full database import: clear any limit set in .profile: moved to inside screen because it must happen within screen to avoid affecting the outer shell
- 01:40 AM Revision 13421: fix: /README.TXT: Full database import: added step to clear any limit set in .profile (applicable to local machine)
- 01:35 AM Revision 13420: bugfix: schemas/Makefile: `%/install: public_.sql`: also need to replace `public` when preceded by ( , as it is in the validations queries
05/07/2014
- 08:14 PM Revision 13419: bugfix: /Makefile: install: use schemas/public/*re*install because schemas/public/install would errexit if the public schema already exists
- 07:17 PM Revision 13418: bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(datasrc), not $(schema), as $schema is only what this var is called in the runscripts
- 06:48 PM Revision 13417: bugfix: inputs/analytical_db/: need dummy table.run file to cause a schema to be created for this datasource
- 06:44 PM Revision 13416: fix: inputs/input.Makefile: $(sortFile): don't print the "add any missing tables to $(sortFile)" message every time the Makefile is run
- 06:44 PM Revision 13415: bugfix: inputs/input.Makefile: install: only run this for datasource dirs
- 05:18 PM Revision 13414: inputs/input.Makefile: install: use ./run's install target for clarity
- 05:18 PM Revision 13413: lib/runscripts/import.run: added install() target
- 04:56 PM Revision 13412: bugfix: inputs/input.Makefile: install: made it idempotent (using new $(datasrc_schema_exists)) so that it could be run by `make install` on an existing system
- 04:02 PM Revision 13411: bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(shell ...)
- 03:31 PM Revision 13410: inputs/input.Makefile: added $(datasrc_schema_exists)
- 03:29 PM Revision 13409: lib/runscripts/in_datasrc_dir.run: use new local.run
- 03:25 PM Revision 13408: added lib/runscripts/local.run
05/06/2014
- 03:39 PM Revision 13407: web/links/index.htm: updated to Firefox bookmarks: iPlant links: fixed broken links
- 03:34 PM Revision 13406: web/links/index.htm: updated to Firefox bookmarks: starscream links: removed favicons so they wouldn't wake the Mac up on network access whenever someone opens the links page
05/03/2014
- 05:30 PM Task #909 (New): in aggregating validations pipeline, support using input result sets from CSVs instead of only by querying the staging tables
- * allows using input query results run on the raw data rather than the preprocessed staging tables
* enables providi... - 04:15 PM Revision 13405: added validation/aggregating/bien3_validation_sequence.xlsx from Brad
- 03:17 PM Revision 13404: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed disabling of enable_hashjoin (the lack of an additional join now makes `SET enable_seqscan = off;` sufficient to prevent a slow hash join)
- 03:10 PM Revision 13403: bugfix: schemas/util.sql: explain2notice_msg_if_can(): don't use util.is_explainable() because the list provided by Postgres (http://www.postgresql.org/docs/9.3/static/sql-explain.html#AEN77691) excludes some query types that are in fact EXPLAIN-able
- 02:03 PM Revision 13402: added inputs/VegBank/verify/outputBien.log.url
- 02:03 PM Revision 13401: inputs/input.Makefile: add: verify/: also svn:ignore *.log
- 12:44 PM Revision 13400: added validation/aggregating/plots/VegBank/bien3_validations_salvias_db_original_adjToVegBank.txt from Mike Lee
- 01:39 AM Revision 13399: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB
- 01:37 AM Revision 13398: schemas/public_.sql: _plots_20_tnrs_input_names, _plots_21_tnrs_output_names: added Brad's comments
- 01:28 AM Revision 13397: bugfix: schemas/util.sql: materialize_query(): contents generated from: need to include search_path so query is runnable
- 01:05 AM Revision 13396: planning/meetings/BIEN conference call availability.xlsx: updated
- 12:47 AM Revision 13395: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed setting of join_collapse_limit (the column now used does not involve an additional join)
05/02/2014
- 11:34 PM Revision 13394: schemas/vegbien.sql: _plots_21_tnrs_output_names: use scrubbed_unique_taxon_name (concatenated output name) because this is more similar to the concatenated name used in _plots_20_tnrs_input_names
- 11:22 PM Revision 13393: bugfix: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: needs to be run with `SET join_collapse_limit = DEFAULT; SET enable_hashjoin = off;` to produce the right query plan
- 10:57 PM Revision 13392: fix: lib/util.py: dict_subset(): raise an error if collections.OrderedDict isn't available, because some callers may depend on this. note that using dict instead of OrderedDict may be the cause of the joining on the wrong columns bug (issue #902).
- 10:42 PM Revision 13391: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB
- 10:01 PM Revision 13390: bugfix: schemas/public_.sql: _plots_21_tnrs_output_names: need to use TNRS output rather than input name
- 09:54 PM Revision 13389: schemas/public_.sql: added _plots_20_tnrs_input_names, _plots_21_tnrs_output_names
- 06:17 PM Revision 13388: schemas/public_.sql: public_validations: schema comment: documented how to run just the output queries on a particular datasource
- 04:53 PM Revision 13387: fix: schemas/public_.sql: rm_all_queries(): also remove output-only result tables
- 04:50 PM Revision 13386: schemas/public_.sql: query_view_regexp(): also match auxiliary tables
- 04:14 PM Revision 13385: schemas/public_.sql: added remake_output_tables(datasource_schema, prefix)
- 03:53 PM Revision 13384: schemas/util.sql: added loop_ignore_errors()
05/01/2014
- 10:38 PM Revision 13383: schemas/public_.sql: added output_queries(prefix)
- 10:27 PM Revision 13382: schemas/public_.sql: validation_views(): added prefix param, which can be used for additional subsetting
- 10:13 PM Revision 13381: schemas/public_.sql: query_relations(): removed no longer used max_prefix_len default param
- 10:05 PM Revision 13380: schemas/public_.sql: query_relations(): use new public_validations.truncated_prefixed_name_regexp(), which does not require passing in a max_prefix_len
- 10:03 PM Revision 13379: schemas/vegbien.sql: added public_validations.truncated_prefixed_name_regexp(), which does not require passing in a max_prefix_len
- 08:55 PM Revision 13378: schemas/public_.sql: rm_output_queries(), validation_views(): use public_validations.query_view_regexp()
- 08:42 PM Revision 13377: schemas/public_.sql: added public_validations.query_view_regexp()
- 08:06 PM Revision 13376: bugfix: schemas/public_.sql: public_validations.rm_output_queries(): default regexp needs to be anchored at beginning of string, and should match the regexp in public_validations.validation_views()
- 01:58 PM Revision 13375: bugfix: inputs/input.Makefile: %/postprocess: invoke runscript if it exists
- 01:37 PM Revision 13374: lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability
- 01:37 PM Revision 13373: bugfix: lib/runscripts/validations.pg.sql.run: updated to reflect that validations.sql is now located inside a subdir, not the datasrc dir
- 01:29 PM Revision 13372: fix: lib/runscripts/file.pg.sql.run: removed include of in_datasrc_dir.run, because this location does not apply to all .sql export scripts
- 01:15 PM Revision 13371: bugfix: inputs/input.Makefile: validations.sql must be in a subdir so it won't get run by sql/install
- 01:11 PM Revision 13370: bugfix: inputs/input.Makefile: validations.sql must be in a subdir so it won't get run by sql/install
- 05:20 AM Revision 13369: inputs/input.Makefile: install: also run validate/install
- 04:44 AM Revision 13368: inputs/input.Makefile: added validate/install
- 04:09 AM Revision 13367: lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability
- 03:22 AM Revision 13366: bugfix: inputs/SALVIAS/validations.sql: need to cast character varying to text so that the types of each side of if() match
- 03:14 AM Revision 13365: bugfix: lib/sh/db.sh: pg_dump(): need use_pg to import $pg_database before checking for existence of $database
- 03:11 AM Revision 13364: lib/sh/util.sh: import_vars: documented that it's idempotent
04/30/2014
- 07:10 PM Revision 13363: /Makefile: python-Linux: removed ordereddict, since OrderedDict is now obtained from `collections`
- 07:06 PM Revision 13362: web/links/index.htm: updated to Firefox bookmarks: fixed broken links
- 06:58 PM Revision 13361: bugfix: lib/util.py: use OrderedDict from collections rather than ordereddict to work with Mac OS X 10.8 Mountain Lion (http://vegpath.org/links/#OrderedDict)
- 06:42 PM Revision 13360: /Makefile: apache-Linux: updated for Ubuntu 14.04, which includes Apache 2.4 in the distribution
- 06:23 PM Revision 13359: /Makefile: install: documented that schemas/public/install also tests that a clean public schema will be installable by full-database import
- 06:12 PM Revision 13358: /Makefile: install: skip the test step at the end because this is not needed for the installation (it can be run separately if desired)
- 05:46 PM Revision 13357: bugfix: **/postprocess.sql: don't use the public schema, because this creates an unsatisfied dependency while the database is being installed, and breaks `make install`
- 05:36 PM Revision 13356: schemas/public_.sql: _km_to_m(): made it a wrapper around util._km_to_m() so this function is only in one place
- 05:32 PM Revision 13355: schemas/util.sql: added _km_to_m() for use when public._km_to_m() is not defined yet
- 02:33 PM Task #908 (New): generate the source-general derived columns directly on the denormalized staging table
- * avoids needing to run the full Python import, which is the part that is currently buggy
* see [[New-style_import#s...
04/29/2014
- 11:36 PM Revision 13354: bugfix: benign_does_not_exist_error(): removed ignore_e=3, because this exit status is also used for other errors
- 11:35 PM Revision 13353: fix: lib/sh/db.sh: benign_does_not_exist_error(): use benign_error=1, which is now supported properly by stderr_matches()
- 11:34 PM Revision 13352: bugfix: lib/sh/util.sh: stderr_matches(): support $benign_error properly, by handling exit status logging in this func instead
- 11:03 PM Revision 13351: bugfix: lib/sh/db.sh: pg_schema_exists(): also need to benignify "does not exist" error if returns false
- 10:42 PM Revision 13350: bugfix: lib/sh/util.sh: stderr_matches(): need to separately display errors that were incorrectly suppressed due to $benign_error
- 10:36 PM Revision 13349: bugfix: lib/sh/util.sh: is_err(): rethrow must be inverted (rethrow->*false* if error)
- 10:32 PM Revision 13348: lib/sh/util.sh: added is_err()
- 09:53 PM Revision 13347: lib/sh/local.sh: public_schema_exists(): moved to lib/sh/db.sh since this no longer depends on BIEN-specific configurations
- 09:42 PM Revision 13346: bugfix: lib/sh/db.sh: public_schema_exists(): don't hide the function call tree so it's clear which function is running the psql commands
- 09:40 PM Revision 13345: bugfix: lib/sh/db.sh: public_schema_exists(): don't hide the function call tree so it's clear which function is running the psql commands
- 09:38 PM Revision 13344: fix: *{.sh,run}: stderr_matches() callers: added benign_error=1 where needed
- 09:36 PM Revision 13343: fix: *{.sh,run}: stderr_matches() callers: usage: documented that they may require benign_error=1
- 09:35 PM Revision 13342: fix: lib/sh/util.sh: stderr_matches(): usage: documented that this may require benign_error=1
- 09:33 PM Revision 13341: fix: /README.TXT: Mac backup: exclude ~/VirtualBox VMs/Ubuntu/Ubuntu.vdi, to avoid it being re-uploaded twice each time, due to an rsync verification error (https://projects.nceas.ucsb.edu/nceas/issues/907)
- 09:23 PM Revision 13340: bugfix: lib/sh/db.sh: pg_cmd(): updated for new echo_vars log_level
- 09:14 PM Revision 13339: fix: lib/sh/db.sh: pg_schema_exists(): display the function name so it's clear which function is running the psql commands
- 05:46 PM Task #907 (Resolved): troubleshoot rsync verification errors
- * normally only happens when a file is concurrently modified, but this file was not in use
* this may indicate disk ... - 05:37 PM Revision 13338: fix: lib/sh/db.sh: pg_schema_exists(): don't use log++ because it hides the command that produces the benign error
- 04:42 PM Revision 13337: /README.TXT: changed "then rerun with l=1 ..." to "then review diff, and rerun with `l=1` prepended" to ensure that user reviews diff before syncing
- 04:40 PM Revision 13336: /README.TXT: to synchronize a Mac's settings with my testing machine's: removed separate step to upload just the VirtualBox VMs, because that is now part of the main upload
- 04:40 PM Revision 13335: fix: /README.TXT: to synchronize a Mac's settings with my testing machine's: need to sync VirtualBox VMs with inplace=1 because they are very large files
- 03:47 PM Revision 13334: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: rsync: documented that this breaks use of ~ to reference the remote user's home dir
- 03:26 PM Revision 13333: /README.TXT: to back up the version history: back up first on the local machine, because often only the svnsync command gets run, and that way it will get backed up immediately to Dropbox (and hourly to Time Machine), while vegbiendev only gets backed up daily to tape
- 03:23 PM Revision 13332: bugfix: /README.TXT: to back up the version history: use absolute path for vegbiendev commands because the Ubuntu 14.04 version of rsync doesn't expand ~ properly
- 02:36 PM Revision 13331: /README.TXT: to back up the version history: use $HOME to make paths platform-independent
- 01:18 PM Revision 13330: bugfix: exports/.rsync_filter.upload: *.csv: need to use `protect` rather than `hide` to prevent overwriting
04/28/2014
04/25/2014
- 11:38 PM Revision 13328: fix: lib/sh/db.sh: psql(): removed debugging changes
- 11:36 PM Revision 13327: bugfix: lib/sh/util.sh: highlight_log_msg(): when not can_highlight_log_msg, need to remove any surrounding formatting
- 11:01 PM Revision 13326: fix: lib/sh/util.sh: die_error_hidden(): always log local vars at same log_level as echo_func
- 10:56 PM Revision 13325: fix: *{.sh,run}: always log kw_params at same log_level as echo_func
- 06:27 PM Revision 13324: lib/sh/util.sh: split_lines(): usage: matched up and synced different syntaxes
- 06:22 PM Revision 13323: bugfix: lib/sh/util.sh: log_msg!(): split_lines does not support being invoked by wrapper; need to use `declare lines; wrapper "split_lines" str` instead
- 06:21 PM Revision 13322: fix: lib/sh/util.sh: split_lines(): usage: documented different syntax for when using wrapper
- 06:09 PM Revision 13321: bugfix: lib/sh/util.sh: die_error_hidden(): echo_func to assist debugging
- 06:07 PM Revision 13320: bugfix: lib/sh/util.sh: split(): need to limit the effects of IFS to just the splitting, so it doesn't cause strange errors in other functions
- 03:20 PM Revision 13319: bugfix: stderr2stdout(): fd 2 *must* be redirected back to fd 2, not log-filtered, in case there are other errors in addition to the benign error. this fixes a bug in pg_schema_exists(), where errors about the DB being down were not displayed because they were log-filtered out.
- 06:15 AM Revision 13318: lib/sh/make.sh: set_make_vars: don't display make vars at verbosity 2 to avoid clutter
- 05:58 AM Task #905 (Resolved): narrow down the cause of the import bug (incorrect join columns and disk space leak)
- _see #887, #902_
h3. -alternate OS approach-
p(. _tried, and problem also occurs on Mac, so using other approac...
04/24/2014
- 05:34 PM Revision 13317: bugfix: lib/sh/make.sh: begin_target: don't echo_func twice
- 05:29 PM Revision 13316: inputs/GBIF/_MySQL/.rsync_ignore: added GBIFPortalDB-*.data.sql.gz, because these are intermediate files
- 05:02 PM Revision 13315: bugfix: /Makefile: $(pg_ctl-Darwin): need to call the command rather than echoing it, as is needed for the Linux version
- 04:59 PM Revision 13314: bugfix: /Makefile: $(pg_ctl-Darwin): need to `cd /` because due to pg_ctl bug, current directory must be accessible by it
- 03:49 PM Revision 13313: bugfix: lib/runscripts/util.run: a non-runscript should have all args passed to main(). this fixes a bug in backups/*_snapshot where "main" would need to be prepended to any args for the script to run correctly.
- 03:43 PM Revision 13312: bugfix: lib/runscripts/util.run: $wrap_fn: invoked script must always run as runscript so that wrapped command is run
- 03:40 PM Revision 13311: lib/runscripts/util.run: added $is_runscript, for use by $wrap_fn
- 03:36 PM Revision 13310: bugfix: lib/runscripts/util.run: $wrap_fn: $top_script doesn't need to be world-executable for most uses of sudo (only if sudoing to non-root)
04/23/2014
- 10:01 PM Revision 13309: bin/in_place: diff: use --brief to avoid scanning the entire file for large files
- 09:57 PM Revision 13308: bin/in_place: added $preserve_mtime flag
- 07:22 PM Task #887: fix disk space leak that fills the disk and crashes the import
- VM upgraded to Ubuntu 14.04 and using the official Ubuntu version of Postgres, but problem still occurs
- 07:16 PM Revision 13307: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: Apache: documented that MultiViews is actually only broken for redirects with the filename "index"
- 07:11 PM Revision 13306: web/.htaccess: for dirs, redirect to index.*: document it is actually just the filename "index" that MultiViews is broken for, other filenames work fine
- 06:06 PM Task #903 (Resolved): fix Ubuntu 14.04 upgrade bug that prevents Apache from displaying vegbiendev.nceas.ucsb.edu properly
- 06:00 PM Task #903: fix Ubuntu 14.04 upgrade bug that prevents Apache from displaying vegbiendev.nceas.ucsb.edu properly
- added workaround for broken MultiViews
- 04:14 PM Task #903 (Resolved): fix Ubuntu 14.04 upgrade bug that prevents Apache from displaying vegbiendev.nceas.ucsb.edu properly
- this consists of 2 problems:
# -http://vegbiendev.nceas.ucsb.edu/index.php now includes the VegCore wiki page inst... - 06:02 PM Revision 13305: bugfix: web/index.php: full directory index: only display if invoked as "vegpath.org/", not "vegpath.org/index.php"
- 05:58 PM Revision 13304: bugfix: web/.htaccess: for dirs, redirect to index.*: added workaround for Ubuntu 14.04, which breaks MultiViews
- 05:56 PM Revision 13303: /Makefile: postgres-Linux: updated to use the official version that comes with Ubuntu 14.04
- 05:53 PM Revision 13302: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: Apache: documented that this breaks MultiViews, so you need to rewrite .htaccess files to avoid using MultiViews
- 05:24 PM Revision 13301: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: added Postgres upgrading instructions
- 05:05 PM Revision 13300: _license/non-open-source/applies_to.txt: Brad: added "anything he created while not working for iPlant, from 2013-7-1..10-31"
- 04:53 PM Revision 13299: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade
- 04:47 PM Revision 13298: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: added phpMyAdmin fixing instructions
- 04:45 PM Revision 13297: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: added phpMyAdmin fixing instructions
- 04:42 PM Task #904 (Resolved): add MySQL public user to allow accessing the normalized VegCore data dictionary
- __* see "normalized VegCore data dictionary":http://vegbiendev.nceas.ucsb.edu/vegbiendev/db/my/VegCore
- 04:31 PM Revision 13296: web/links/index.htm: updated to Firefox bookmarks: Ubuntu: Ubuntu 14.04 upgrade: added things broken by it. PostgreSQL: fixed links.
- 04:00 PM Task #884 (Rejected): fix Postgres bug that causes query planner to use seq scans and slow sorts instead of index scans in the import
- duplicate of #902: slow sorts are caused by joining on the wrong columns, not query planner settings
- 03:52 PM Task #902 (Resolved): fix bug that causes joining on the wrong columns in the import
- _bug fixed in r14074_
h3. issue
* in some queries, the columns being joined on are completely the wrong set (co... - 03:16 PM Task #901: schedule regular pg_dump backups of the DB
- see @backups/pg_snapshot@, @backups/mysql_snapshot@
- 01:08 PM Revision 13295: /Makefile: postgres-Linux: added warning that the install commands were designed to run on Ubuntu 12.04, which is no longer the version used by vegbiendev (it is now 14.04)
- 12:09 PM Revision 13294: backups/mysql_snapshot: documented initial vegbiendev->jupiter upload time for GBIF/raw_occurrence_record.MYD (7 h for 91 GB = 3.7 MB/s)
- 12:12 AM Revision 13293: fix: web/links/index.htm: updated to Firefox bookmarks: PostgreSQL: backups: wal_keep_segments method: clarified how to determine the value of wal_keep_segments. filesystem-level backups: documented the advantages of filesystem-level backups over traditional database-level backups with pg_dump.
04/22/2014
- 11:32 PM Revision 13292: fix: web/links/index.htm: updated to Firefox bookmarks: PostgreSQL: backups: wal_keep_segments: restored annotations
- 11:26 PM Revision 13291: web/links/index.htm: updated to Firefox bookmarks: PostgreSQL: backups: documented how to set up online and offline backups (with two possible approaches for online backups)
- 11:24 PM Revision 13290: web/links/index.htm: updated to Firefox bookmarks: PostgreSQL: backups: documented how to set up online and offline backups (with two possible approaches for online backups)
- 11:21 PM Revision 13289: web/links/index.htm: updated to Firefox bookmarks: PostgreSQL: backups: documented how to set up online and offline backups (with two possible approaches for online backups)
- 10:33 PM Revision 13288: lib/sh/db.sh: pg_snapshot(): perform online backup if possible, to avoid needing to restart the server
- 10:31 PM Revision 13287: lib/sh/db.sh: added pg_start_backup(), pg_stop_backup()
- 10:29 PM Revision 13286: lib/sh/db.sh: psql(): only set redirs if can redirect
- 10:17 PM Revision 13285: bugfix: psql(): when as_root is on, need to avoid redirections as these are not passed through by sudo
- 08:14 PM Revision 13284: /README.TXT: use `sudo -u ... -i` instead of `sudo su - ...` to avoid using two commands to accomplish the login
- 06:01 PM Revision 13283: bugfix: lib/sh/db.sh: psql(): don't use `--file /dev/fd/40` when can't redirect
- 05:59 PM Revision 13282: fix: lib/sh/db.sh: psql(): when using sudo with custom $stdin, raise error that this is not supported
- 05:50 PM Revision 13281: lib/sh/db.sh: psql(): $data_bypasses_filter: renamed to $data2stdout for clarity
- 05:29 PM Revision 13280: lib/sh/db.sh: psql(): $bypass_ok: renamed to $can_redir for clarity
- 05:22 PM Revision 13279: lib/sh/db.sh: psql(): usage: $stdin: documented that can also use process substitution for this
- 04:45 PM Revision 13278: bugfix: lib/sh/util.sh: `type` calls: need -- before cmd in case it starts with -
- 04:44 PM Revision 13277: lib/sh/util.sh: cmd2rel_path: use is_extern()
- 04:00 PM Revision 13276: lib/sh/sync.sh: db_snapshot(): use `end_try` as specified in `try` usage
- 03:59 PM Revision 13275: lib/sh/util.sh: try: usage: added location of finally block
- 03:57 PM Revision 13274: schemas/postgresql.conf: wal_level: set to hot_standby to enable online backup with pg_start_backup()
- 03:04 PM Revision 13273: lib/sh/sync.sh: upload(): always print the function and kw_params
- 04:00 AM Revision 13272: added backups/mysql_snapshot, pg_snapshot
04/21/2014
- 08:21 PM Revision 13271: bugfix: lib/sh/util.sh: type(): need to handle options before command name
- 08:21 PM Revision 13270: lib/sh/util.sh: added 1st_non_opt()
- 08:07 PM Revision 13269: lib/sh/util.sh: unalias(): use self_builtin, which is now defined before it
- 08:01 PM Revision 13268: lib/runscripts/util.run: sudo(): avoid slow $wrap_fn when using `command` (ie. always executable)
- 07:59 PM Revision 13267: lib/sh/util.sh: unalias(): use self_builtin, which is now defined before it
- 07:57 PM Revision 13266: bugfix: lib/sh/util.sh: commands: `type` calls: need to account for the fact that any alias is already expanded
- 07:56 PM Revision 13265: lib/sh/util.sh: functions: moved before commands since commands are more complex
- 07:38 PM Revision 13264: lib/sh/sync.sh: db_copy() and callers: pass args as rsync options
- 07:28 PM Revision 13263: fix: lib/sh/sync.sh: db_copy(): need to exclude files which prevent tape backup
- 07:22 PM Revision 13262: lib/sh/db.sh: added pg_ctl(), pg_snapshot()
- 07:17 PM Revision 13261: lib/sh/sync.sh: db_snapshot(): copy changes before stopping DB to minimize the time that it's shut down
- 07:12 PM Revision 13260: lib/sh/sync.sh: db_snapshot(): factored copy operation out into separate db_copy() function
- 07:03 PM Revision 13259: lib/sh/db.sh: mysql_snapshot(): use new db_snapshot()
- 07:02 PM Revision 13258: lib/sh/sync.sh: added db_snapshot()
- 05:55 PM Revision 13257: lib/Firefox_bookmarks.reformat.csv: changed "page's own description" to "page's self-description" for clarity
- 05:50 PM Revision 13256: web/links/index.htm: updated to Firefox bookmarks: removed dead links
- 05:43 PM Revision 13255: web/links/index.htm: updated to Firefox bookmarks: updated favicons
- 05:28 PM Revision 13254: web/links/index.htm: updated to Firefox bookmarks: modifying a running shell script: updated to document that `svn up` actually *does* use two-stage save automatically
- 04:46 PM Revision 13253: lib/sh/db.sh: mysql_snapshot(): for large files, don't re-copy entire file
- 04:44 PM Revision 13252: lib/sh/db.sh: mysql_snapshot(): use live mode as the default
- 04:30 PM Revision 13251: fix: lib/sh/db.sh: mysql_snapshot(): need to create dest dir if doesn't exist
- 04:27 PM Revision 13250: bugfix: lib/sh/db.sh: mysql_snapshot(): try: need to use split syntax with prep_try instead, to work with prefix vars
- 04:23 PM Revision 13249: bugfix: lib/sh/db.sh: mysql_snapshot(): try: need to use split syntax with prep_try instead, to work with prefix vars
- 04:20 PM Revision 13248: fix: lib/sh/util.sh: try usage: documented that the split syntax with prep_try is meant to be used with vars before the cmd
- 03:37 PM Revision 13247: fix: lib/sh/util.sh: echo_vars(): also need to print unset vars (including unset kw_params)
- 03:31 PM Revision 13246: lib/sh/util.sh: echo_vars(): put loop var on same line as `for`
- 02:59 PM Revision 13245: bugfix: lib/sh/util.sh: sudo(): need to preserve PATH separately because -E does not preserve this
- 02:17 PM Revision 13244: lib/sh/util.sh: echo_redirs_cmd(): inline the function alias since it's only used in one place
- 02:15 PM Revision 13243: bugfix: lib/sh/util.sh: redir(): need to load new aliases before it
- 02:13 PM Revision 13242: lib/sh/util.sh: echo_redirs_cmd(): log $PATH to facilitate troubleshooting
- 01:54 PM Revision 13241: lib/sh/util.sh: echo_redirs_cmd(): documented what the $(...) section does
- 01:50 PM Revision 13240: lib/sh/util.sh: echo_redirs_cmd(): moved comment about <>file redirs to line that it applies to
- 01:47 PM Revision 13239: lib/sh/util.sh: moved echo_redirs_cmd() to right before redir() which uses it
- 02:55 AM Revision 13238: lib/sh/util.sh: catch(): log at higher log_level, since this is internal code
- 02:43 AM Revision 13237: fix: lib/sh/util.sh: die_e(): treat SIGPIPE as benign error
- 02:32 AM Revision 13236: lib/sh/util.sh: removed no longer used ignore_sig(). use ignore() instead, which now supports SIG*.
- 02:32 AM Revision 13235: lib/sh/util.sh: piped_cmd(): use ignore, which now supports SIG*
- 02:31 AM Revision 13234: lib/sh/util.sh: signals: catch(): added echo_func
- 02:28 AM Revision 13233: lib/sh/util.sh: set_global_fds(): debug to global stderr in case stderr filtered
- 02:26 AM Revision 13232: lib/sh/util.sh: debugging: use configurable debug_fd (set to $err_fd)
- 02:13 AM Revision 13231: lib/sh/util.sh: signals: override catch() to support SIG* as exception type
- 02:11 AM Revision 13230: lib/sh/util.sh: moved primitives sections before more complex sections that depend on them
- 02:07 AM Revision 13229: lib/sh/util.sh: 2nd functions section: moved to 1st functions section
- 01:16 AM Revision 13228: bugfix: lib/sh/util.sh: added workaround for bash bug where exit sometimes inxeplicably ignores $?
- 01:15 AM Revision 13227: fix: lib/sh/util.sh: self_builtin: avoid $() so that $? isn't modified
- 01:07 AM Revision 13226: lib/sh/util.sh: use new self_builtin
- 01:06 AM Revision 13225: lib/sh/util.sh: added self_builtin
- 12:50 AM Revision 13224: lib/sh/util.sh: pv(), pf(): moved to debugging section
- 12:48 AM Revision 13223: bugfix: lib/sh/util.sh: stderr_matches(): also need to handle any filter error, such as caused by Ctrl+C
- 12:26 AM Revision 13222: lib/sh/util.sh: stderr_matches(): echo_vars @PIPESTATUS_ to assist debugging
04/20/2014
- 11:37 PM Revision 13221: bugfix: lib/sh/util.sh: stderr2stdout(): use piped_cmd to ignore SIGPIPE since the output of this will piped to another command
- 06:22 PM Revision 13220: lib/sh/util.sh: setup_log_fd(): $log_fd: use 3 (stdlog) since other scripts are likely to use this for logging as well
- 06:20 PM Revision 13219: fix: lib/sh/util.sh: setup_log_fd(): fd_set_default(): use $log_fd instead of repeating the value of it
- 06:05 PM Revision 13218: lib/sh/util.sh: die(): log at higher log_level, since this is logging code
- 06:02 PM Revision 13217: lib/sh/util.sh: highlight_log_msg(): log at higher log_level, since this is logging code
- 05:54 PM Revision 13216: bugfix: lib/runscripts/util.run: $subdirs: adjusted log_level now that echo_vars is one log_level lower
- 05:46 PM Revision 13215: bugfix: lib/sh/util.sh: stderr_matches(): only set benign_error=1 if the matched error occurred
- 05:44 PM Revision 13214: lib/sh/util.sh: ignore_e(): also set benign_error=1
- 05:40 PM Revision 13213: fix: lib/sh/util.sh: prep_try alias: removed inaccurate comment
- 05:33 PM Revision 13212: bugfix: lib/sh/util.sh: stdout2fd(): moved after redir() which it depends on
- 05:24 PM Revision 13211: fix: lib/sh/util.sh: command(): moved `|| die_e` to command__exec so it would be properly indented under the echoed command
- 05:12 PM Revision 13210: lib/sh/util.sh: verbosity_compat(): log at higher log_level because it's logging code
- 05:10 PM Revision 13209: lib/sh/util.sh: $benign_error: log at higher log_level because it's logging code
- 05:06 PM Revision 13208: lib/runscripts/util.run: $wrap_fn: log at higher log_level because it's startup code
- 04:55 PM Revision 13207: lib/sh/util.sh: $top_* vars, $is_outermost: log at higher log_level because it's startup code
- 04:52 PM Revision 13206: lib/sh/util.sh: $top_script: echo_vars this like the other $top_* vars
- 04:50 PM Revision 13205: lib/sh/util.sh: .(): log at higher log_level because it's startup code
- 04:45 PM Revision 13204: lib/sh/util.sh: is_dot_script(): run with higher log_level since this is run at the beginning of the script
- 04:44 PM Revision 13203: lib/sh/util.sh, runscripts/util.run: set_paths(): run with higher log_level to hide all the paths that are set at the beginning of the script
- 04:28 PM Revision 13202: lib/sh/util.sh: added log++ stub
- 04:22 PM Revision 13201: lib/sh/util.sh: added log_local stub
- 03:15 PM Revision 13200: lib/sh/util.sh: added log() stub so internal commands can use it
- 03:10 PM Revision 13199: fix: lib/sh/util.sh: echo_vars(): log at same log_level as echo_func so kw_params are displayed along with positional params
- 03:08 PM Revision 13198: fix: lib/sh/util.sh: rel_path(): log this internal command at a higher log_level so it's normally hidden
- 02:43 PM Revision 13197: fix: lib/sh/util.sh: log_msg!(): log split_lines at a higher log_level so it's normally hidden
04/19/2014
- 10:22 PM Revision 13196: bugfix: lib/sh/util.sh: stderr_matches(): `log_local; log++` should apply to just stdout_contains() and part of stderr2stdout() rather than all of stderr_matches()
- 10:14 PM Revision 13195: inputs/Madidi/_src/: set svn:ignore
- 10:13 PM Revision 13194: added backups/vegbien.r13002.backup.md5, vegbien.r13160.backup.md5
- 10:12 PM Revision 13193: backups/TNRS.backup.md5: updated
- 10:09 PM Revision 13192: lib/sh/util.sh: stderr_matches(): run at higher log_level because error-handling internals should not be logged by default
- 10:07 PM Revision 13191: bugfix: lib/sh/db.sh: mysql_ctl(): need to ignore errors if not running
- 10:04 PM Revision 13190: bugfix: lib/sh/util.sh: stderr_matches(): handle any error: only ignore_e if the error exit status was associated with the matched error message
- 09:57 PM Revision 13189: bugfix: lib/sh/util.sh: stderr_matches(): handle any error: need force-exit with rethrow_exit() because caller's test of return status disables errexit
- 09:54 PM Revision 13188: lib/sh/util.sh: added rethrow_exit(), which exits even where errexit is disabled
- 09:48 PM Revision 13187: bugfix: lib/sh/db.sh: mysql_snapshot(): need to run `mysql_ctl start` even if there is an error
- 09:38 PM Revision 13186: lib/sh/db.sh: mysql_snapshot(): $to: default to $from.bak
- 08:06 PM Revision 13185: lib/sh/db.sh: added mysql_snapshot()
- 07:46 PM Revision 13184: lib/sh/db.sh: added mysql_ctl()
- 07:35 PM Revision 13183: lib/sh/db.sh: pg_cmd(): updated to use new sudo()
- 07:14 PM Revision 13182: lib/runscripts/util.run: added sudo() override that uses $wrap_fn to support shell functions
- 07:13 PM Revision 13181: fix: lib/runscripts/util.run: $wrap_fn: make it usable even if $top_script isn't world-executable
- 07:11 PM Revision 13180: lib/sh/util.sh: sudo alias: use function instead so this can be overridden
- 07:09 PM Revision 13179: lib/sh/util.sh: added is_intern()
- 07:07 PM Revision 13178: lib/sh/util.sh: is_callable(): use just $1 because multiple args are not applicable
- 07:06 PM Revision 13177: lib/sh/util.sh: added is_world_executable()
- 07:06 PM Revision 13176: lib/sh/util.sh: added has_perms()
- 06:49 PM Revision 13175: lib/sh/util.sh: esc_args(): renamed to just esc() because this can also be used on a single value
- 05:51 PM Revision 13174: lib/sh/util.sh: added is_extern()
- 10:57 AM Revision 13173: lib/sh/util.sh: added sudo alias to alias-expand command
- 10:57 AM Revision 13172: lib/sh/db.sh: pg_cmd(): $as_root: use $sudo
- 10:54 AM Revision 13171: lib/sh/util.sh: added $sudo
- 10:24 AM Revision 13170: lib/sh/util.sh: added cp alias
- 09:55 AM Revision 13169: lib/sh/db.sh: removed no longer used pg_as_root(), which was buggy anyway. use `as_root=1 ...` instead.
- 09:38 AM Revision 13168: lib/sh/db.sh: mysql_ANSI: fixed comment
04/18/2014
- 06:57 PM Revision 13167: added backups/users.sql.run
- 05:34 PM Revision 13166: lib/sh/db.sh: pg_dump(): support dumping entire cluster, and cluster users
- 05:10 PM Revision 13165: lib/sh/db.sh: pg_cmd(): added $as_root switch
04/17/2014
- 08:21 PM Revision 13164: fix: inputs/SALVIAS/projects/postprocess.sql: remove private data that should not be publicly visible: preserve datasets with ipr_specific = '', because they *are* actually redistributable, according to Brad (http://wiki.vegpath.org/2014-04-17_conference_call#conditions-of-use)
- 08:14 PM Task #887: fix disk space leak that fills the disk and crashes the import
- main DB backed up, (close to?) ready to roll back and/or upgrade the VM
- 10:31 AM Task #887: fix disk space leak that fills the disk and crashes the import
- submitted support request to restore vegbiendev to last working configuration and install a past revision of Postgres...
- 08:12 PM Task #901: schedule regular pg_dump backups of the DB
- seem to have settled on shutting down VM before tape backup as the approach for this
- 01:23 PM Task #901 (New): schedule regular pg_dump backups of the DB
- * this is not backed up with the rest of the VM due to bandwidth limitations and available tape drive space
* use @-... - 05:28 PM Revision 13163: web/links/index.htm: updated to Firefox bookmarks: PostgreSQL: added operator classes. added backups: filesystem-level backup, continuous archiving, WAL logging, etc. virtual collaboration: updated annotations.
- 02:52 PM Revision 13162: lib/sh/db.sh pg_dump(), bin/pg_dump_vegbien: --format=plain: removed comment that this is the plain format, because this is now self-documenting
- 02:51 PM Revision 13161: lib/sh/db.sh pg_dump(), bin/pg_dump_vegbien: --format: use the long form of the formats to make the code self-documenting
- 03:44 AM Revision 13160: validation/aggregating/specimens/qualitative_validations_specimens.sql: updated to DB
- 03:41 AM Revision 13159: validation/aggregating/specimens/NY/qualitative_validations_source_db_NYBG.VegCore.sql: updated to inputs/NY/validations.sql
- 03:32 AM Revision 13158: schemas/vegbien.ERD.mwb: regenerated exports
- 03:31 AM Revision 13157: fix: lib/PostgreSQL-MySQL.csv: preserve schema assignments by translating `SET search_path` to `USE`
- 02:54 AM Revision 13156: schemas/vegbien.ERD.mwb: regenerated exports
- 02:53 AM Revision 13155: schemas/vegbien.ERD.mwb: added geoscrub, TNRS tables, as requested in the 2014-04-10 conference call (wiki.vegpath.org/2014-04-10_conference_call#VegBIEN-schema)
- 02:39 AM Revision 13154: schemas/Makefile: vegbien.sql: also include geoscrub, TNRS schemas, as requested in the 2014-04-10 conference call (wiki.vegpath.org/2014-04-10_conference_call#VegBIEN-schema). this involves having a separate public_.sql file for restoring the public schema.
04/16/2014
- 11:02 PM Revision 13153: schemas/vegbien.ERD.mwb: regenerated exports
- 10:49 PM Revision 13152: bugfix: inputs/NY/validations.sql: _specimens_07_list_of_verbatim_subspecific_taxa_with_author: updated filter condition to match output query
- 10:48 PM Revision 13151: inputs/NY/run: `make inputs/NY/validate`: updated runtime (8 min, with added queries)
- 10:24 PM Revision 13150: fix: inputs/NY/Ecatalog_all/map.csv, postprocess.sql: remapped substrate, vegetation to locationRemarks
- 10:14 PM Task #899 (New): remove dependencies on Mac
- * avoids needing to support Mac as well as Linux in all our scripts
** note that Mac software must be installed manu... - 10:10 PM Task #898 (New): remove dependencies on the development machine
- * the development process should not require both a VM and a specially-configured local machine to make changes to th...
- 06:41 PM Revision 13149: fix: inputs/NY/Ecatalog_all/map.csv, postprocess.sql: remapped substrate, vegetation to locationRemarks
- 06:35 PM Revision 13148: bugfix: lib/runscripts/import.run: all(): also need to propagate $rm to import()
- 04:24 PM Revision 13147: bugfix: inputs/NY/validations.sql, schemas/vegbien.sql: _specimens_13*: also need to include coordinate pairs which have one of their coordinates NULL, by using OR instead of AND
- 04:15 PM Revision 13146: bugfix: inputs/NY/validations.sql: _specimens_13b_list_of_all_decimal_lat_long: matched column types to output query
- 04:14 PM Revision 13145: bugfix: inputs/NY/validations.sql: _specimens_13a_list_of_all_verbatim_lat_long: matched column types to output query
- 03:13 PM Revision 13144: inputs/NY/validations.sql, schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: added breakdowns _specimens_13a_list_of_all_verbatim_lat_long, _specimens_13b_list_of_all_decimal_lat_long to help troubleshoot the diff
- 02:04 PM Revision 13143: fix: inputs/NY/validations.sql, schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: count lat/longs together instead of separately, because the DISTINCT is by coordinate *pair*, not individual coordinate value (which wouldn't make much sense)
04/15/2014
- 08:12 PM Revision 13142: bugfix: schemas/vegbien.sql: rm_output_queries(): need to account for the fact that util.truncated_prefixed_name_regexp() returns a whole-string regexp. this drops support for removing output queries with a particular group prefix, which we no longer use.
- 07:59 PM Revision 13141: bugfix: schemas/vegbien.sql: rm_output_queries(): need to include relations whose names were truncated, as well
- 07:14 PM Revision 13140: fix: schemas/vegbien.sql: public_validations schema comment: to remove a validations query so its columns can be changed: use rm_output_queries() rather than rm_query_view() because that also removes input queries
- 07:00 PM Revision 13139: bugfix: schemas/util.sql: is_castable(): need to pass NULL through, for proper NULL propagation
- 06:52 PM Revision 13138: fix: inputs/NY/validations.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: use new is_castable(), which is much more accurate than Brad's custom regexp for determining if something is numeric
- 06:29 PM Revision 13137: inputs/NY/validations.-.util.sql: added util.is_castable() wrapper
- 06:12 PM Revision 13136: schemas/util.sql: added is_castable()
- 06:10 PM Revision 13135: schemas/util.sql: added try_cast()
- 05:51 PM Revision 13134: schemas/util.sql: added util.cast(), which allows casting to an arbitrary type without eval()
04/14/2014
- 05:04 PM Revision 13133: bugfix: schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: DISTINCT: added coordsaccuracy_m
- 05:02 PM Revision 13132: bugfix: schemas/vegbien.sql: coordinates_unique: added coordsaccuracy_m
- 04:56 PM Revision 13131: fix: schemas/vegbien.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: need to DISTINCT the values that are being counted, because the coordinates_unique unique constraint includes other columns as well, so there may be multiple instances of each lat/long
- 04:51 PM Revision 13130: bugfix: inputs/NY/validations.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: need to include both lat and long in the value to DISTINCT on
- 04:48 PM Revision 13129: fix: inputs/NY/validations.sql: _specimens_13_count_of_all_verbatim_and_decimal_lat_long: need to DISTINCT the values that are being counted, because they are merged by the coordinates_unique unique constraint in the import
- 04:24 PM Revision 13128: validation/aggregating/pipeline/aggregating_validations_pipeline.odg: diff tables: integrated row labels into table
- 04:04 PM Revision 13127: validation/aggregating/pipeline/aggregating_validations_pipeline.odg: diff tables: added line for different rows (vs. missing/extra)
- 03:58 PM Revision 13126: inputs/NY/run: `make inputs/NY/validate`: documented slow queries: _specimens_12_distinct_collector_name_collect_num_date_w_count
- 03:23 PM Revision 13125: inputs/SALVIAS/run_: `make inputs/SALVIAS/validate`: documented slow queries (_plots_06a_list_of_stems). these may need to have their query plans rechecked.
- 03:22 PM Revision 13124: inputs/NY/run, inputs/SALVIAS/run_: `make inputs/.../validate`: updated runtime (+2 min)
Also available in: Atom