Project

General

Profile

Activity

From 04/23/2014 to 05/22/2014

05/22/2014

04:43 PM Task #932 (New): Discovering BIEN data through iPlant

iPlant will assign DOIs to datasets made public through its infrastructure. For the BIEN db data, the plan is:
* t...
Martha Narro
04:32 PM Task #931 (New): BIEN db at iPlant
Parent issue for tasks related to moving BIEN3 relational database and associated analytical views to iPlant. Martha Narro
04:20 PM Revision 13520: bugfix: config/VirtualBox_VMs/: switched from symlinks to hard links, because svn does not follow symlinks
Aaron Marcuse-Kubitza
02:07 PM Revision 13519: added config/VirtualBox_VMs/, containing the .vbox settings and password.txt (a non-empty password is needed for some system commands)
Aaron Marcuse-Kubitza

05/21/2014

07:30 PM Revision 13518: inputs/.TNRS/schema.sql: MatchedTaxon_modify(): removed no longer needed DROP VIEW statement
Aaron Marcuse-Kubitza
07:30 PM Revision 13517: schemas/util.sql: recreate(): perform the correct DROP VIEW in the function itself so that the caller does not have to worry about forming it properly
Aaron Marcuse-Kubitza
07:19 PM Revision 13516: bugfix: schemas/util.sql: mk_drop_from_create(): added `DROP`
Aaron Marcuse-Kubitza
07:18 PM Revision 13515: schemas/util.sql: added mk_drop_from_create()
Aaron Marcuse-Kubitza
06:52 PM Revision 13514: schemas/util.sql: added regexp_match()
Aaron Marcuse-Kubitza
05:58 PM Revision 13513: planning/meetings/BIEN conference call availability.xlsx: updated
Aaron Marcuse-Kubitza
05:50 PM Revision 13512: fix: schemas/util.sql: force_recreate(): renamed to just recreate(), because "force" normally implies that things will be deleted, which this function does not do
Aaron Marcuse-Kubitza
05:49 PM Revision 13511: fix: schemas/public_.sql: _plots_20_tnrs_names: fixed order of tnrs_taxonomic_status, tnrs_accepted_name_family
Aaron Marcuse-Kubitza
04:45 PM Revision 13510: schemas/public_.sql: _plots_20_tnrs_names: added tnrs_taxonomic_status, as requested in issue #915
Aaron Marcuse-Kubitza
04:34 PM Revision 13509: schemas/public_.sql: analytical_stem_view: taxonomic_status: don't filter with map_taxonomic_status() anymore since this is now done by "TNRS"."MatchedTaxon"
Aaron Marcuse-Kubitza
04:25 PM Revision 13508: fix: inputs/.TNRS/schema.sql: MatchedTaxon.taxonomicStatus: filter using map_taxonomic_status() so that the corrected value is available in the normalized DB, not just analytical_stem
Aaron Marcuse-Kubitza
04:05 PM Revision 13507: inputs/.TNRS/schema.sql: MatchedTaxon: to modify: use new MatchedTaxon_modify(), which eliminates the work of putting together the dependent views
Aaron Marcuse-Kubitza
03:53 PM Revision 13506: inputs/.TNRS/schema.sql: added MatchedTaxon_modify()
Aaron Marcuse-Kubitza
03:44 PM Revision 13505: schemas/util.sql: show_create_view(): use new mk_set_relation_metadata()
Aaron Marcuse-Kubitza
03:41 PM Revision 13504: schemas/util.sql: added mk_set_relation_metadata()
Aaron Marcuse-Kubitza
04:13 AM Revision 13503: bugfix: inputs/.TNRS/schema.sql: map_taxonomic_status(): need to use accepted name instead of scrubbed name (which also includes no-opinion names), as described at http://wiki.vegpath.org/2013-11-14_conference_call#taxonomic-fields. this used to be the accepted name, but got switched when the concatenated name was also used to store the matched name for no-opinion names.
Aaron Marcuse-Kubitza
01:50 AM Revision 13502: schemas/public_.sql: _plots_20_tnrs_names: use accepted_morphospecies_binomial, which is now available directly in the TNRS results, instead of just the morphospecies_suffix, as requested in issue #915
Aaron Marcuse-Kubitza
01:27 AM Revision 13501: inputs/.TNRS/schema.sql: MatchedTaxon: documented how to modify it (using util.force_recreate())
Aaron Marcuse-Kubitza
01:23 AM Revision 13500: fix: lib/PostgreSQL-MySQL.csv: also remove left-behind lines such as `$$);`
Aaron Marcuse-Kubitza
12:59 AM Revision 13499: schemas/util.sql: force_recreate(): documented usage, which is somewhat complex and has several important parts
Aaron Marcuse-Kubitza

05/20/2014

05:46 PM Revision 13498: inputs/.TNRS/schema.sql: MatchedTaxon, etc.: added accepted_morphospecies_binomial derived field
Aaron Marcuse-Kubitza
05:40 PM Revision 13497: bugfix: schemas/util.sql: restore_views(): need to specify view name for a manual existence check, in case the view definition becomes invalid, which would produce a nonstandard (uncatchable) exception
Aaron Marcuse-Kubitza
05:33 PM Revision 13496: bugfix: schemas/util.sql: explain(): don't display any log messages printed by IMMUTABLE functions in the query, which EXPLAIN apparently runs
Aaron Marcuse-Kubitza
05:21 PM Revision 13495: schemas/util.sql: create_if_not_exists(): added relation param, which can be used to ensure that a standard exception is always generated if the relation exists, even if the table definition would be invalid (which generates a variety of exceptions)
Aaron Marcuse-Kubitza
04:26 PM Task #929 (New): change TNRS client to store metadata
* see details in [[2013-06-13 conference call#include-TNRS-version-and-settings-in-TNRS-cache]]
* after implementi...
Aaron Marcuse-Kubitza
04:17 PM Task #928 (New): switch to new TNRS setup
# #927 (part of #915)
# #917
# #916 (using rescrubbed names)
# #915
# -#929 (metadata)- _postponed_
Aaron Marcuse-Kubitza
03:26 PM Task #927 (Resolved): move morphospecies name formation to normalized DB
* currently, it's only in the analytical_stem view Aaron Marcuse-Kubitza
02:39 PM Revision 13494: /README.TXT: to back up the vegbiendev databases: added step to review diff before performing backup, in case the backup scripts break and try to synchronize things incorrectly
Aaron Marcuse-Kubitza

05/19/2014

09:08 PM Revision 13493: bugfix: schemas/util.sql: try_cast(), is_castable(): also need to handle syntax_error_or_access_rule_violation, which is thrown by ::regclass
Aaron Marcuse-Kubitza
08:52 PM Revision 13492: schemas/util.sql: added relation_exists()
Aaron Marcuse-Kubitza
08:27 PM Revision 13491: schemas/util.sql: save_drop_views()/restore_views(): store the view path in addition to the definition so it can be used in restoring
Aaron Marcuse-Kubitza
08:24 PM Revision 13490: bugfix: schemas/util.sql: in_reverse(): can't use `SELECT *` for composite types because this expands them
Aaron Marcuse-Kubitza
08:21 PM Revision 13489: bugfix: schemas/util.sql: array_reverse(): can't use `SELECT *` for composite types because this expands them
Aaron Marcuse-Kubitza
07:50 PM Revision 13488: schemas/util.sql: save_drop_views()/restore_views(): use opaque return type so it can be changed without changing the local var type in functions that use this
Aaron Marcuse-Kubitza
07:22 PM Revision 13487: schemas/util.sql: force_recreate(): recreate_users_cmds: renamed to restore_views_info since this is now whatever is needed by util.restore_views()
Aaron Marcuse-Kubitza
07:16 PM Revision 13486: schemas/util.sql: added restore_views() and use it in force_recreate()
Aaron Marcuse-Kubitza
02:07 AM Revision 13485: bugfix: schemas/util.sql: save_drop_views(): views must be *dropped* in reverse dependency order, but *returned* in forward dependency order
Aaron Marcuse-Kubitza
01:36 AM Revision 13484: bugfix: schemas/util.sql: force_recreate(): dependent views need to be re-created in *forward* dependency order. this is done by reversing the dependencies only for save_drop_views().
Aaron Marcuse-Kubitza
12:48 AM Revision 13483: bugfix: schemas/util.sql: mk_set_comment(): need to use VIEW instead of TABLE for views
Aaron Marcuse-Kubitza
12:29 AM Revision 13482: bugfix: schemas/util.sql: show_create_view(): also need to include view comment
Aaron Marcuse-Kubitza
12:24 AM Revision 13481: fix: schemas/util.sql: mk_set_comment(): added ;
Aaron Marcuse-Kubitza
12:20 AM Revision 13480: fix: schemas/util.sql: show_set_comment(): don't display a COMMENT ON statement if no comment, because this will be appended to table defs, etc. and would create clutter
Aaron Marcuse-Kubitza
12:16 AM Revision 13479: bugfix: schemas/util.sql: mk_set_comment(): need to handle NULL comment properly
Aaron Marcuse-Kubitza
12:06 AM Revision 13478: schemas/util.sql: added show_set_comment(regclass)
Aaron Marcuse-Kubitza

05/18/2014

11:57 PM Revision 13477: schemas/util.sql: set_comment(): use util.mk_set_comment()
Aaron Marcuse-Kubitza
11:48 PM Revision 13476: schemas/Makefile: added vegbien/install, vegbien/uninstall which also handle TNRS
Aaron Marcuse-Kubitza

05/17/2014

06:26 PM Revision 13475: fix: schemas/Makefile: vegbien.sql: also remake inputs/.TNRS/schema.sql since its contents change along with vegbien.sql
Aaron Marcuse-Kubitza
06:22 PM Revision 13474: schemas/util.sql: added mk_set_comment()
Aaron Marcuse-Kubitza
06:22 PM Revision 13473: bugfix: lib/runscripts/util.run: $is_runscript: unexport so don't pass it to invoked scripts
Aaron Marcuse-Kubitza
06:01 PM Revision 13472: backups/pg_snapshot: documented runtime for initial backup (4 h)
Aaron Marcuse-Kubitza
05:36 PM Revision 13471: bugfix: schemas/util.sql: force_recreate(): recreate_users_cmds: use create_if_not_exists() rather than eval(), because cmd might manually re-create a deleted dependent view, causing it to already exist
Aaron Marcuse-Kubitza
05:27 PM Revision 13470: bugfix: schemas/util.sql: save_drop_view(): util.show_create_view(): don't include `OR REPLACE` because that causes nonuniform errors (eg. invalid_table_definition), instead of the standard duplicate_table exception caught by util.create_if_not_exists()
Aaron Marcuse-Kubitza
05:25 PM Revision 13469: fix: schemas/util.sql: show_create_view(): removed extra trailing ; because pg_get_viewdef() now includes one
Aaron Marcuse-Kubitza
05:24 PM Revision 13468: schemas/util.sql: show_create_view(): made inclusion of `OR REPLACE` configurable
Aaron Marcuse-Kubitza
05:10 PM Revision 13467: schemas/util.sql: save_drop_view(): use util.drop_view()
Aaron Marcuse-Kubitza
03:40 PM Revision 13466: /README.TXT: Maintenance: added steps to back up the vegbiendev databases
Aaron Marcuse-Kubitza
02:15 PM Revision 13465: bugfix: lib/runscripts/util.run: run_args_cmd(): don't prepend main to args if no args, because for a non-runscript, all args will be passed to main(), leading `main` to be doubled
Aaron Marcuse-Kubitza
01:30 PM Revision 13464: lib/tnrs.py: use the TNRS dev server (with private URL in tnrs.url) instead of the live server, since that contains datasources that we need
Aaron Marcuse-Kubitza
01:29 PM Revision 13463: lib/streams.py: added file_get_contents()
Aaron Marcuse-Kubitza
01:14 PM Revision 13462: lib/tnrs.py: configure the server separately from the base URL
Aaron Marcuse-Kubitza
01:12 PM Revision 13461: lib/: svn:ignore tnrs.url so the TNRS dev server URL does not become public
Aaron Marcuse-Kubitza

05/16/2014

12:03 AM Revision 13460: bugfix: web/.phpPgAdmin/.htaccess: only append subject if non-empty, to avoid confusing redirect.php
Aaron Marcuse-Kubitza

05/15/2014

11:57 PM Revision 13459: bugfix: web/.phpPgAdmin/.htaccess: custom pages: need to handle in query string because .php filename would be treated as path element
Aaron Marcuse-Kubitza
11:32 PM Revision 13458: web/.phpPgAdmin/.htaccess: support custom .php pages
Aaron Marcuse-Kubitza
10:30 AM Revision 13457: planning/meetings/BIEN conference call availability.xlsx: updated
Aaron Marcuse-Kubitza
09:43 AM Task #917: TNRS: Instructions for new version with TPL
Aaron, for now (next couple of months, this must be done on the development TNRS web app since TPL is only on the dev... Martha Narro

05/13/2014

02:32 PM Task #917 (New): TNRS: Instructions for new version with TPL
From Brad:
I’ve given some thought to the TPL matter. The algorithm isn’t hard, but Aaron will have to do the sort...
Martha Narro
11:19 AM Revision 13456: schemas/util.sql: save_drop_views(): return text[] instead of concatenated text so that the queries will be run individually and be easier to debug
Aaron Marcuse-Kubitza
11:00 AM Revision 13455: schemas/util.sql: added eval(queries text[])
Aaron Marcuse-Kubitza
09:43 AM Revision 13454: bugfix: schemas/util.sql: force_recreate(): need to drop users in reverse to avoid dependencies that block the drop
Aaron Marcuse-Kubitza
09:25 AM Revision 13453: schemas/util.sql: added array_reverse()
Aaron Marcuse-Kubitza
09:25 AM Revision 13452: schemas/util.sql: added in_reverse()
Aaron Marcuse-Kubitza
08:39 AM Revision 13451: bugfix: schemas/util.sql: force_recreate(): need to use [[:lower:]], not [:lower:]
Aaron Marcuse-Kubitza
08:34 AM Revision 13450: schemas/util.sql: force_recreate(): debug-print PG_EXCEPTION_DETAIL, users to assist in debugging
Aaron Marcuse-Kubitza
08:33 AM Revision 13449: fix: schemas/util.sql: debug_print_value(): added trailing newline to visually distinguish the printed value
Aaron Marcuse-Kubitza
08:27 AM Revision 13448: schemas/util.sql: added debug_print_var()
Aaron Marcuse-Kubitza
07:06 AM Revision 13447: schemas/util.sql: debug_print_return_value(): use new util.debug_print_value()
Aaron Marcuse-Kubitza
06:59 AM Revision 13446: schemas/util.sql: added debug_print_value()
Aaron Marcuse-Kubitza
05:45 AM Revision 13445: fix: schemas/util.sql: util.force_recreate(): also support dropping things besides tables
Aaron Marcuse-Kubitza
04:50 AM Revision 13444: inputs/.TNRS/schema.sql: MatchedTaxon.Accepted_name_species: mapped to accepted_species_binomial
Aaron Marcuse-Kubitza
04:09 AM Revision 13443: fix: inputs/.TNRS/schema.sql: COMMENTs: always include newline before and after
Aaron Marcuse-Kubitza
03:50 AM Revision 13442: bugfix: schemas/public_.sql: _plots_20_tnrs_names: tnrs_accepted_*: use new accepted_*, *not* scrubbed_*, as scrubbed_* does *not* contain only the accepted name (as implied by the tnrs_accepted_* column name)
Aaron Marcuse-Kubitza
03:46 AM Revision 13441: bugfix: inputs/.TNRS/schema.sql: taxon_scrub, etc.: undid rename of accepted name columns to scrubbed_* (r13435), because these are actually *not* the same (scrubbed_* is the combination of accepted and no-opinion names). the accepted name columns will now be named accepted_*, following the standard naming scheme.
Aaron Marcuse-Kubitza
03:28 AM Revision 13440: bugfix: schemas/public_.sql: _plots_20_tnrs_names: morphospeciesSuffix must still be called morphospecies_suffix, not taxon_morphospecies, because that column has not been updated to the taxon_morphospecies yet (fixes r13435)
Aaron Marcuse-Kubitza
03:13 AM Revision 13439: fix: inputs/.TNRS/schema.sql: taxon_scrub, etc.: scrubbed_*: use columns from MatchedTaxon whenever possible, to as much as possible avoid the need to join to taxon_scrub.scrubbed_unique_taxon_name.*
Aaron Marcuse-Kubitza
02:53 AM Revision 13438: schemas/Makefile: vegbien.sql: exclude the same set of Source tables excluded by inputs/.TNRS/schema.sql, so that reinstalling TNRS doesn't change the contents of this file
Aaron Marcuse-Kubitza
02:29 AM Revision 13437: bugfix: inputs/.TNRS/grants.sql: added GRANT statements from schema.sql because these aren't run by `make inputs/.TNRS/reinstall`
Aaron Marcuse-Kubitza

05/12/2014

07:06 PM Revision 13436: lib/tnrs.py: retrieval_request_template: taxonomic_constraint, source_sorting: documented their meaning and why they need to be on/off
Aaron Marcuse-Kubitza
05:03 PM Task #916: Taxon name validation: VegBank
regarding:
Line 617: TNRS gives a synonym for Aronia prunifolia in a different genus, but this is missed here
L...
Martha Narro
04:35 PM Task #916 (New): Taxon name validation: VegBank
Hi Aaron,
Bob only had time to get part way through the VegBank taxon validation file you sent, but there are some...
Martha Narro
04:28 PM Revision 13435: schemas/public_.sql: _plots_20_tnrs_names: changed to set of columns requested by Martha
Aaron Marcuse-Kubitza
03:13 PM Task #915 (New): Taxon Name Validation
1. Please correct the following in taxonMorphospecies:
1.1. -Family should be included in taxonMorphospecies only ...
Martha Narro

05/10/2014

09:32 PM Revision 13434: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB
Aaron Marcuse-Kubitza
09:17 PM Revision 13433: schemas/public_.sql: _plots_20_tnrs_names: added morphospecies suffix in order to also validate it
Aaron Marcuse-Kubitza
09:13 PM Revision 13432: schemas/public_.sql: _plots_20_tnrs_names: added verbatim name in order to also validate the formation of the concatenated name
Aaron Marcuse-Kubitza
08:49 PM Revision 13431: schemas/public_.sql: merged _plots_20_tnrs_input_names, _plots_21_tnrs_output_names into _plots_20_tnrs_names so the input and output names can be compared side-by-side
Aaron Marcuse-Kubitza
08:10 PM Revision 13430: schemas/vegbien.sql: synced to DB, which adds Source.errors table indexes added in the import
Aaron Marcuse-Kubitza

05/09/2014

07:27 PM Revision 13429: /README.TXT: Full database import: for test import, turn off DB backup: added () after after_import to show that it's a function
Aaron Marcuse-Kubitza
07:24 PM Revision 13428: /README.TXT: Full database import: refer to the shell instead of screen because it's also possible to use a nested shell for this
Aaron Marcuse-Kubitza
02:51 AM Revision 13427: bugfix: /README.TXT: Full database import: to import just a subset of the datasources: need to set @inputs separately from declaring it, because declare does not support () on Mac
Aaron Marcuse-Kubitza
02:16 AM Revision 13426: fix: /README.TXT: Full database import: on local machine: need to allow logging output to go to log files
Aaron Marcuse-Kubitza
02:05 AM Revision 13425: /README.TXT: Full database import: for test import, added step to turn off DB backup
Aaron Marcuse-Kubitza
01:48 AM Revision 13424: fix: /README.TXT: Full database import: setting $version: moved to inside screen to avoid affecting the outer shell
Aaron Marcuse-Kubitza
01:46 AM Revision 13423: /README.TXT: Full database import: screen: added alternative for a small import (just use nested shell)
Aaron Marcuse-Kubitza
01:42 AM Revision 13422: fix: /README.TXT: Full database import: clear any limit set in .profile: moved to inside screen because it must happen within screen to avoid affecting the outer shell
Aaron Marcuse-Kubitza
01:40 AM Revision 13421: fix: /README.TXT: Full database import: added step to clear any limit set in .profile (applicable to local machine)
Aaron Marcuse-Kubitza
01:35 AM Revision 13420: bugfix: schemas/Makefile: `%/install: public_.sql`: also need to replace `public` when preceded by ( , as it is in the validations queries
Aaron Marcuse-Kubitza

05/07/2014

08:14 PM Revision 13419: bugfix: /Makefile: install: use schemas/public/*re*install because schemas/public/install would errexit if the public schema already exists
Aaron Marcuse-Kubitza
07:17 PM Revision 13418: bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(datasrc), not $(schema), as $schema is only what this var is called in the runscripts
Aaron Marcuse-Kubitza
06:48 PM Revision 13417: bugfix: inputs/analytical_db/: need dummy table.run file to cause a schema to be created for this datasource
Aaron Marcuse-Kubitza
06:44 PM Revision 13416: fix: inputs/input.Makefile: $(sortFile): don't print the "add any missing tables to $(sortFile)" message every time the Makefile is run
Aaron Marcuse-Kubitza
06:44 PM Revision 13415: bugfix: inputs/input.Makefile: install: only run this for datasource dirs
Aaron Marcuse-Kubitza
05:18 PM Revision 13414: inputs/input.Makefile: install: use ./run's install target for clarity
Aaron Marcuse-Kubitza
05:18 PM Revision 13413: lib/runscripts/import.run: added install() target
Aaron Marcuse-Kubitza
04:56 PM Revision 13412: bugfix: inputs/input.Makefile: install: made it idempotent (using new $(datasrc_schema_exists)) so that it could be run by `make install` on an existing system
Aaron Marcuse-Kubitza
04:02 PM Revision 13411: bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(shell ...)
Aaron Marcuse-Kubitza
03:31 PM Revision 13410: inputs/input.Makefile: added $(datasrc_schema_exists)
Aaron Marcuse-Kubitza
03:29 PM Revision 13409: lib/runscripts/in_datasrc_dir.run: use new local.run
Aaron Marcuse-Kubitza
03:25 PM Revision 13408: added lib/runscripts/local.run
Aaron Marcuse-Kubitza

05/06/2014

03:39 PM Revision 13407: web/links/index.htm: updated to Firefox bookmarks: iPlant links: fixed broken links
Aaron Marcuse-Kubitza
03:34 PM Revision 13406: web/links/index.htm: updated to Firefox bookmarks: starscream links: removed favicons so they wouldn't wake the Mac up on network access whenever someone opens the links page
Aaron Marcuse-Kubitza

05/03/2014

05:30 PM Task #909 (New): in aggregating validations pipeline, support using input result sets from CSVs instead of only by querying the staging tables
* allows using input query results run on the raw data rather than the preprocessed staging tables
* enables providi...
Aaron Marcuse-Kubitza
04:15 PM Revision 13405: added validation/aggregating/bien3_validation_sequence.xlsx from Brad
Aaron Marcuse-Kubitza
03:17 PM Revision 13404: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed disabling of enable_hashjoin (the lack of an additional join now makes `SET enable_seqscan = off;` sufficient to prevent a slow hash join)
Aaron Marcuse-Kubitza
03:10 PM Revision 13403: bugfix: schemas/util.sql: explain2notice_msg_if_can(): don't use util.is_explainable() because the list provided by Postgres (http://www.postgresql.org/docs/9.3/static/sql-explain.html#AEN77691) excludes some query types that are in fact EXPLAIN-able
Aaron Marcuse-Kubitza
02:03 PM Revision 13402: added inputs/VegBank/verify/outputBien.log.url
Aaron Marcuse-Kubitza
02:03 PM Revision 13401: inputs/input.Makefile: add: verify/: also svn:ignore *.log
Aaron Marcuse-Kubitza
12:44 PM Revision 13400: added validation/aggregating/plots/VegBank/bien3_validations_salvias_db_original_adjToVegBank.txt from Mike Lee
Aaron Marcuse-Kubitza
01:39 AM Revision 13399: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB
Aaron Marcuse-Kubitza
01:37 AM Revision 13398: schemas/public_.sql: _plots_20_tnrs_input_names, _plots_21_tnrs_output_names: added Brad's comments
Aaron Marcuse-Kubitza
01:28 AM Revision 13397: bugfix: schemas/util.sql: materialize_query(): contents generated from: need to include search_path so query is runnable
Aaron Marcuse-Kubitza
01:05 AM Revision 13396: planning/meetings/BIEN conference call availability.xlsx: updated
Aaron Marcuse-Kubitza
12:47 AM Revision 13395: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: removed no longer needed setting of join_collapse_limit (the column now used does not involve an additional join)
Aaron Marcuse-Kubitza

05/02/2014

11:34 PM Revision 13394: schemas/vegbien.sql: _plots_21_tnrs_output_names: use scrubbed_unique_taxon_name (concatenated output name) because this is more similar to the concatenated name used in _plots_20_tnrs_input_names
Aaron Marcuse-Kubitza
11:22 PM Revision 13393: bugfix: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: _plots_21_tnrs_output_names: needs to be run with `SET join_collapse_limit = DEFAULT; SET enable_hashjoin = off;` to produce the right query plan
Aaron Marcuse-Kubitza
10:57 PM Revision 13392: fix: lib/util.py: dict_subset(): raise an error if collections.OrderedDict isn't available, because some callers may depend on this. note that using dict instead of OrderedDict may be the cause of the joining on the wrong columns bug (issue #902).
Aaron Marcuse-Kubitza
10:42 PM Revision 13391: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB
Aaron Marcuse-Kubitza
10:01 PM Revision 13390: bugfix: schemas/public_.sql: _plots_21_tnrs_output_names: need to use TNRS output rather than input name
Aaron Marcuse-Kubitza
09:54 PM Revision 13389: schemas/public_.sql: added _plots_20_tnrs_input_names, _plots_21_tnrs_output_names
Aaron Marcuse-Kubitza
06:17 PM Revision 13388: schemas/public_.sql: public_validations: schema comment: documented how to run just the output queries on a particular datasource
Aaron Marcuse-Kubitza
04:53 PM Revision 13387: fix: schemas/public_.sql: rm_all_queries(): also remove output-only result tables
Aaron Marcuse-Kubitza
04:50 PM Revision 13386: schemas/public_.sql: query_view_regexp(): also match auxiliary tables
Aaron Marcuse-Kubitza
04:14 PM Revision 13385: schemas/public_.sql: added remake_output_tables(datasource_schema, prefix)
Aaron Marcuse-Kubitza
03:53 PM Revision 13384: schemas/util.sql: added loop_ignore_errors()
Aaron Marcuse-Kubitza

05/01/2014

10:38 PM Revision 13383: schemas/public_.sql: added output_queries(prefix)
Aaron Marcuse-Kubitza
10:27 PM Revision 13382: schemas/public_.sql: validation_views(): added prefix param, which can be used for additional subsetting
Aaron Marcuse-Kubitza
10:13 PM Revision 13381: schemas/public_.sql: query_relations(): removed no longer used max_prefix_len default param
Aaron Marcuse-Kubitza
10:05 PM Revision 13380: schemas/public_.sql: query_relations(): use new public_validations.truncated_prefixed_name_regexp(), which does not require passing in a max_prefix_len
Aaron Marcuse-Kubitza
10:03 PM Revision 13379: schemas/vegbien.sql: added public_validations.truncated_prefixed_name_regexp(), which does not require passing in a max_prefix_len
Aaron Marcuse-Kubitza
08:55 PM Revision 13378: schemas/public_.sql: rm_output_queries(), validation_views(): use public_validations.query_view_regexp()
Aaron Marcuse-Kubitza
08:42 PM Revision 13377: schemas/public_.sql: added public_validations.query_view_regexp()
Aaron Marcuse-Kubitza
08:06 PM Revision 13376: bugfix: schemas/public_.sql: public_validations.rm_output_queries(): default regexp needs to be anchored at beginning of string, and should match the regexp in public_validations.validation_views()
Aaron Marcuse-Kubitza
01:58 PM Revision 13375: bugfix: inputs/input.Makefile: %/postprocess: invoke runscript if it exists
Aaron Marcuse-Kubitza
01:37 PM Revision 13374: lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability
Aaron Marcuse-Kubitza
01:37 PM Revision 13373: bugfix: lib/runscripts/validations.pg.sql.run: updated to reflect that validations.sql is now located inside a subdir, not the datasrc dir
Aaron Marcuse-Kubitza
01:29 PM Revision 13372: fix: lib/runscripts/file.pg.sql.run: removed include of in_datasrc_dir.run, because this location does not apply to all .sql export scripts
Aaron Marcuse-Kubitza
01:15 PM Revision 13371: bugfix: inputs/input.Makefile: validations.sql must be in a subdir so it won't get run by sql/install
Aaron Marcuse-Kubitza
01:11 PM Revision 13370: bugfix: inputs/input.Makefile: validations.sql must be in a subdir so it won't get run by sql/install
Aaron Marcuse-Kubitza
05:20 AM Revision 13369: inputs/input.Makefile: install: also run validate/install
Aaron Marcuse-Kubitza
04:44 AM Revision 13368: inputs/input.Makefile: added validate/install
Aaron Marcuse-Kubitza
04:09 AM Revision 13367: lib/runscripts/validations.pg.sql.run: export_(): make the export idempotent for easier re-runnability
Aaron Marcuse-Kubitza
03:22 AM Revision 13366: bugfix: inputs/SALVIAS/validations.sql: need to cast character varying to text so that the types of each side of if() match
Aaron Marcuse-Kubitza
03:14 AM Revision 13365: bugfix: lib/sh/db.sh: pg_dump(): need use_pg to import $pg_database before checking for existence of $database
Aaron Marcuse-Kubitza
03:11 AM Revision 13364: lib/sh/util.sh: import_vars: documented that it's idempotent
Aaron Marcuse-Kubitza

04/30/2014

07:10 PM Revision 13363: /Makefile: python-Linux: removed ordereddict, since OrderedDict is now obtained from `collections`
Aaron Marcuse-Kubitza
07:06 PM Revision 13362: web/links/index.htm: updated to Firefox bookmarks: fixed broken links
Aaron Marcuse-Kubitza
06:58 PM Revision 13361: bugfix: lib/util.py: use OrderedDict from collections rather than ordereddict to work with Mac OS X 10.8 Mountain Lion (http://vegpath.org/links/#OrderedDict)
Aaron Marcuse-Kubitza
06:42 PM Revision 13360: /Makefile: apache-Linux: updated for Ubuntu 14.04, which includes Apache 2.4 in the distribution
Aaron Marcuse-Kubitza
06:23 PM Revision 13359: /Makefile: install: documented that schemas/public/install also tests that a clean public schema will be installable by full-database import
Aaron Marcuse-Kubitza
06:12 PM Revision 13358: /Makefile: install: skip the test step at the end because this is not needed for the installation (it can be run separately if desired)
Aaron Marcuse-Kubitza
05:46 PM Revision 13357: bugfix: **/postprocess.sql: don't use the public schema, because this creates an unsatisfied dependency while the database is being installed, and breaks `make install`
Aaron Marcuse-Kubitza
05:36 PM Revision 13356: schemas/public_.sql: _km_to_m(): made it a wrapper around util._km_to_m() so this function is only in one place
Aaron Marcuse-Kubitza
05:32 PM Revision 13355: schemas/util.sql: added _km_to_m() for use when public._km_to_m() is not defined yet
Aaron Marcuse-Kubitza
02:33 PM Task #908 (New): generate the source-general derived columns directly on the denormalized staging table
* avoids needing to run the full Python import, which is the part that is currently buggy
* see [[New-style_import#s...
Aaron Marcuse-Kubitza

04/29/2014

11:36 PM Revision 13354: bugfix: benign_does_not_exist_error(): removed ignore_e=3, because this exit status is also used for other errors
Aaron Marcuse-Kubitza
11:35 PM Revision 13353: fix: lib/sh/db.sh: benign_does_not_exist_error(): use benign_error=1, which is now supported properly by stderr_matches()
Aaron Marcuse-Kubitza
11:34 PM Revision 13352: bugfix: lib/sh/util.sh: stderr_matches(): support $benign_error properly, by handling exit status logging in this func instead
Aaron Marcuse-Kubitza
11:03 PM Revision 13351: bugfix: lib/sh/db.sh: pg_schema_exists(): also need to benignify "does not exist" error if returns false
Aaron Marcuse-Kubitza
10:42 PM Revision 13350: bugfix: lib/sh/util.sh: stderr_matches(): need to separately display errors that were incorrectly suppressed due to $benign_error
Aaron Marcuse-Kubitza
10:36 PM Revision 13349: bugfix: lib/sh/util.sh: is_err(): rethrow must be inverted (rethrow->*false* if error)
Aaron Marcuse-Kubitza
10:32 PM Revision 13348: lib/sh/util.sh: added is_err()
Aaron Marcuse-Kubitza
09:53 PM Revision 13347: lib/sh/local.sh: public_schema_exists(): moved to lib/sh/db.sh since this no longer depends on BIEN-specific configurations
Aaron Marcuse-Kubitza
09:42 PM Revision 13346: bugfix: lib/sh/db.sh: public_schema_exists(): don't hide the function call tree so it's clear which function is running the psql commands
Aaron Marcuse-Kubitza
09:40 PM Revision 13345: bugfix: lib/sh/db.sh: public_schema_exists(): don't hide the function call tree so it's clear which function is running the psql commands
Aaron Marcuse-Kubitza
09:38 PM Revision 13344: fix: *{.sh,run}: stderr_matches() callers: added benign_error=1 where needed
Aaron Marcuse-Kubitza
09:36 PM Revision 13343: fix: *{.sh,run}: stderr_matches() callers: usage: documented that they may require benign_error=1
Aaron Marcuse-Kubitza
09:35 PM Revision 13342: fix: lib/sh/util.sh: stderr_matches(): usage: documented that this may require benign_error=1
Aaron Marcuse-Kubitza
09:33 PM Revision 13341: fix: /README.TXT: Mac backup: exclude ~/VirtualBox VMs/Ubuntu/Ubuntu.vdi, to avoid it being re-uploaded twice each time, due to an rsync verification error (https://projects.nceas.ucsb.edu/nceas/issues/907)
Aaron Marcuse-Kubitza
09:23 PM Revision 13340: bugfix: lib/sh/db.sh: pg_cmd(): updated for new echo_vars log_level
Aaron Marcuse-Kubitza
09:14 PM Revision 13339: fix: lib/sh/db.sh: pg_schema_exists(): display the function name so it's clear which function is running the psql commands
Aaron Marcuse-Kubitza
05:46 PM Task #907 (Resolved): troubleshoot rsync verification errors
* normally only happens when a file is concurrently modified, but this file was not in use
* this may indicate disk ...
Aaron Marcuse-Kubitza
05:37 PM Revision 13338: fix: lib/sh/db.sh: pg_schema_exists(): don't use log++ because it hides the command that produces the benign error
Aaron Marcuse-Kubitza
04:42 PM Revision 13337: /README.TXT: changed "then rerun with l=1 ..." to "then review diff, and rerun with `l=1` prepended" to ensure that user reviews diff before syncing
Aaron Marcuse-Kubitza
04:40 PM Revision 13336: /README.TXT: to synchronize a Mac's settings with my testing machine's: removed separate step to upload just the VirtualBox VMs, because that is now part of the main upload
Aaron Marcuse-Kubitza
04:40 PM Revision 13335: fix: /README.TXT: to synchronize a Mac's settings with my testing machine's: need to sync VirtualBox VMs with inplace=1 because they are very large files
Aaron Marcuse-Kubitza
03:47 PM Revision 13334: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: rsync: documented that this breaks use of ~ to reference the remote user's home dir
Aaron Marcuse-Kubitza
03:26 PM Revision 13333: /README.TXT: to back up the version history: back up first on the local machine, because often only the svnsync command gets run, and that way it will get backed up immediately to Dropbox (and hourly to Time Machine), while vegbiendev only gets backed up daily to tape
Aaron Marcuse-Kubitza
03:23 PM Revision 13332: bugfix: /README.TXT: to back up the version history: use absolute path for vegbiendev commands because the Ubuntu 14.04 version of rsync doesn't expand ~ properly
Aaron Marcuse-Kubitza
02:36 PM Revision 13331: /README.TXT: to back up the version history: use $HOME to make paths platform-independent
Aaron Marcuse-Kubitza
01:18 PM Revision 13330: bugfix: exports/.rsync_filter.upload: *.csv: need to use `protect` rather than `hide` to prevent overwriting
Aaron Marcuse-Kubitza

04/28/2014

11:11 PM Revision 13329: fix: exports/.rsync_filter.upload: *.csv: don't allow test runs' exports to overwrite the backed up ones
Aaron Marcuse-Kubitza

04/25/2014

11:38 PM Revision 13328: fix: lib/sh/db.sh: psql(): removed debugging changes
Aaron Marcuse-Kubitza
11:36 PM Revision 13327: bugfix: lib/sh/util.sh: highlight_log_msg(): when not can_highlight_log_msg, need to remove any surrounding formatting
Aaron Marcuse-Kubitza
11:01 PM Revision 13326: fix: lib/sh/util.sh: die_error_hidden(): always log local vars at same log_level as echo_func
Aaron Marcuse-Kubitza
10:56 PM Revision 13325: fix: *{.sh,run}: always log kw_params at same log_level as echo_func
Aaron Marcuse-Kubitza
06:27 PM Revision 13324: lib/sh/util.sh: split_lines(): usage: matched up and synced different syntaxes
Aaron Marcuse-Kubitza
06:22 PM Revision 13323: bugfix: lib/sh/util.sh: log_msg!(): split_lines does not support being invoked by wrapper; need to use `declare lines; wrapper "split_lines" str` instead
Aaron Marcuse-Kubitza
06:21 PM Revision 13322: fix: lib/sh/util.sh: split_lines(): usage: documented different syntax for when using wrapper
Aaron Marcuse-Kubitza
06:09 PM Revision 13321: bugfix: lib/sh/util.sh: die_error_hidden(): echo_func to assist debugging
Aaron Marcuse-Kubitza
06:07 PM Revision 13320: bugfix: lib/sh/util.sh: split(): need to limit the effects of IFS to just the splitting, so it doesn't cause strange errors in other functions
Aaron Marcuse-Kubitza
03:20 PM Revision 13319: bugfix: stderr2stdout(): fd 2 *must* be redirected back to fd 2, not log-filtered, in case there are other errors in addition to the benign error. this fixes a bug in pg_schema_exists(), where errors about the DB being down were not displayed because they were log-filtered out.
Aaron Marcuse-Kubitza
06:15 AM Revision 13318: lib/sh/make.sh: set_make_vars: don't display make vars at verbosity 2 to avoid clutter
Aaron Marcuse-Kubitza
05:58 AM Task #905 (Resolved): narrow down the cause of the import bug (incorrect join columns and disk space leak)
_see #887, #902_
h3. -alternate OS approach-
p(. _tried, and problem also occurs on Mac, so using other approac...
Aaron Marcuse-Kubitza

04/24/2014

05:34 PM Revision 13317: bugfix: lib/sh/make.sh: begin_target: don't echo_func twice
Aaron Marcuse-Kubitza
05:29 PM Revision 13316: inputs/GBIF/_MySQL/.rsync_ignore: added GBIFPortalDB-*.data.sql.gz, because these are intermediate files
Aaron Marcuse-Kubitza
05:02 PM Revision 13315: bugfix: /Makefile: $(pg_ctl-Darwin): need to call the command rather than echoing it, as is needed for the Linux version
Aaron Marcuse-Kubitza
04:59 PM Revision 13314: bugfix: /Makefile: $(pg_ctl-Darwin): need to `cd /` because due to pg_ctl bug, current directory must be accessible by it
Aaron Marcuse-Kubitza
03:49 PM Revision 13313: bugfix: lib/runscripts/util.run: a non-runscript should have all args passed to main(). this fixes a bug in backups/*_snapshot where "main" would need to be prepended to any args for the script to run correctly.
Aaron Marcuse-Kubitza
03:43 PM Revision 13312: bugfix: lib/runscripts/util.run: $wrap_fn: invoked script must always run as runscript so that wrapped command is run
Aaron Marcuse-Kubitza
03:40 PM Revision 13311: lib/runscripts/util.run: added $is_runscript, for use by $wrap_fn
Aaron Marcuse-Kubitza
03:36 PM Revision 13310: bugfix: lib/runscripts/util.run: $wrap_fn: $top_script doesn't need to be world-executable for most uses of sudo (only if sudoing to non-root)
Aaron Marcuse-Kubitza

04/23/2014

10:01 PM Revision 13309: bin/in_place: diff: use --brief to avoid scanning the entire file for large files
Aaron Marcuse-Kubitza
09:57 PM Revision 13308: bin/in_place: added $preserve_mtime flag
Aaron Marcuse-Kubitza
07:22 PM Task #887: fix disk space leak that fills the disk and crashes the import
VM upgraded to Ubuntu 14.04 and using the official Ubuntu version of Postgres, but problem still occurs Aaron Marcuse-Kubitza
07:16 PM Revision 13307: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: Apache: documented that MultiViews is actually only broken for redirects with the filename "index"
Aaron Marcuse-Kubitza
07:11 PM Revision 13306: web/.htaccess: for dirs, redirect to index.*: document it is actually just the filename "index" that MultiViews is broken for, other filenames work fine
Aaron Marcuse-Kubitza
06:06 PM Task #903 (Resolved): fix Ubuntu 14.04 upgrade bug that prevents Apache from displaying vegbiendev.nceas.ucsb.edu properly
Aaron Marcuse-Kubitza
06:00 PM Task #903: fix Ubuntu 14.04 upgrade bug that prevents Apache from displaying vegbiendev.nceas.ucsb.edu properly
added workaround for broken MultiViews Aaron Marcuse-Kubitza
04:14 PM Task #903 (Resolved): fix Ubuntu 14.04 upgrade bug that prevents Apache from displaying vegbiendev.nceas.ucsb.edu properly
this consists of 2 problems:
# -http://vegbiendev.nceas.ucsb.edu/index.php now includes the VegCore wiki page inst...
Aaron Marcuse-Kubitza
06:02 PM Revision 13305: bugfix: web/index.php: full directory index: only display if invoked as "vegpath.org/", not "vegpath.org/index.php"
Aaron Marcuse-Kubitza
05:58 PM Revision 13304: bugfix: web/.htaccess: for dirs, redirect to index.*: added workaround for Ubuntu 14.04, which breaks MultiViews
Aaron Marcuse-Kubitza
05:56 PM Revision 13303: /Makefile: postgres-Linux: updated to use the official version that comes with Ubuntu 14.04
Aaron Marcuse-Kubitza
05:53 PM Revision 13302: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: Apache: documented that this breaks MultiViews, so you need to rewrite .htaccess files to avoid using MultiViews
Aaron Marcuse-Kubitza
05:24 PM Revision 13301: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: added Postgres upgrading instructions
Aaron Marcuse-Kubitza
05:05 PM Revision 13300: _license/non-open-source/applies_to.txt: Brad: added "anything he created while not working for iPlant, from 2013-7-1..10-31"
Aaron Marcuse-Kubitza
04:53 PM Revision 13299: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade
Aaron Marcuse-Kubitza
04:47 PM Revision 13298: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: added phpMyAdmin fixing instructions
Aaron Marcuse-Kubitza
04:45 PM Revision 13297: web/links/index.htm: updated to Firefox bookmarks: Ubuntu 14.04 upgrade: added phpMyAdmin fixing instructions
Aaron Marcuse-Kubitza
04:42 PM Task #904 (Resolved): add MySQL public user to allow accessing the normalized VegCore data dictionary
__* see "normalized VegCore data dictionary":http://vegbiendev.nceas.ucsb.edu/vegbiendev/db/my/VegCore Aaron Marcuse-Kubitza
04:31 PM Revision 13296: web/links/index.htm: updated to Firefox bookmarks: Ubuntu: Ubuntu 14.04 upgrade: added things broken by it. PostgreSQL: fixed links.
Aaron Marcuse-Kubitza
04:00 PM Task #884 (Rejected): fix Postgres bug that causes query planner to use seq scans and slow sorts instead of index scans in the import
duplicate of #902: slow sorts are caused by joining on the wrong columns, not query planner settings Aaron Marcuse-Kubitza
03:52 PM Task #902 (Resolved): fix bug that causes joining on the wrong columns in the import
_bug fixed in r14074_
h3. issue
* in some queries, the columns being joined on are completely the wrong set (co...
Aaron Marcuse-Kubitza
03:16 PM Task #901: schedule regular pg_dump backups of the DB
see @backups/pg_snapshot@, @backups/mysql_snapshot@ Aaron Marcuse-Kubitza
01:08 PM Revision 13295: /Makefile: postgres-Linux: added warning that the install commands were designed to run on Ubuntu 12.04, which is no longer the version used by vegbiendev (it is now 14.04)
Aaron Marcuse-Kubitza
12:09 PM Revision 13294: backups/mysql_snapshot: documented initial vegbiendev->jupiter upload time for GBIF/raw_occurrence_record.MYD (7 h for 91 GB = 3.7 MB/s)
Aaron Marcuse-Kubitza
12:12 AM Revision 13293: fix: web/links/index.htm: updated to Firefox bookmarks: PostgreSQL: backups: wal_keep_segments method: clarified how to determine the value of wal_keep_segments. filesystem-level backups: documented the advantages of filesystem-level backups over traditional database-level backups with pg_dump.
Aaron Marcuse-Kubitza
 

Also available in: Atom