Activity
From 06/26/2014 to 07/25/2014
07/25/2014
- 09:57 PM Revision 14339: fix: schemas/util.sql: copy_data(): also need to ANALYZE table after loading data into it
- 09:13 PM Revision 14338: schemas/public_.sql: analytical_stem_view: use viewFullOccurrence_individual_view to avoid duplicated code between the views. this moves the stem-related columns to the end of the view.
- 06:26 PM Revision 14337: schemas/public_.sql: added viewFullOccurrence_individual_view
- 06:20 PM Revision 14336: web/username_prefix.php: substitutions: also support "" because Postgres doubles " to escape them
- 06:17 PM Revision 14335: web/username_prefix.php: added support for substitutions ( x."_y@url -> x.x_y@url ), which can be used when a table name is repeated in a column name
- 06:15 PM Revision 14334: backups/TNRS.backup.md5: updated
- 04:09 PM Bug #943 (New): in TNRS, the Constrain by Source option is applied incorrectly to the results download
- h3. issue
from "TNRS issue 188":https://pods.iplantcollaborative.org/jira/browse/TNRS-188 (in iPlant JIRA):
> t... - 02:31 PM Revision 14333: schemas/util.sql: derived_col_constraint_initially_enabled(): clarified that this is always based on whether the existing constraint is enabled
- 02:21 PM Revision 14332: inputs/.TNRS/schema.sql: taxon_match: COMMENT: added steps to port derived column changes to vegbiendev
- 02:12 PM Revision 14331: schemas/util.sql: added derived_cols_export(), which creates SQL to port the derived cols to another DB
- 12:41 PM Revision 14330: bugfix: inputs/.TNRS/schema.sql: taxon_match: derived columns: use new "matched~Name[_no_author]___@TNRS__@vegpath.org" instead of "*Name_matched" so that "No suitable matches found." is removed before concatenating with other fields
- 12:14 PM Revision 14329: inputs/.TNRS/schema.sql: taxon_match: added derived column "matched~Name[_no_author]___@TNRS__@vegpath.org", which removes the "No suitable matches found." string
07/22/2014
- 06:50 AM Revision 14328: inputs/.TNRS/schema.sql: reordered derived columns in dependency order
- 06:46 AM Revision 14327: bugfix: inputs/.TNRS/schema.sql: "[accepted_]morphospecies[_binomial]__@Brad__.TNRS@vegpath.org": need to use "[accepted_]genus__@DwC__@vegpath.org" rather than "*Accepted_name" for this for rank = genus
- 06:44 AM Revision 14326: inputs/.TNRS/schema.sql: taxon_match: added derived column "[scrubbed_]morphospecies[_binomial]__@Brad__.TNRS@vegpath.org"
- 06:41 AM Revision 14325: bugfix: inputs/.TNRS/schema.sql: "[accepted_]genus__@DwC__@vegpath.org": need to populate this for rank = genus
- 05:47 AM Revision 14324: inputs/.TNRS/schema.sql: taxon_match: added derived column "[scrubbed_]taxonomicStatus__@DwC__@vegpath.org"
- 05:30 AM Revision 14323: bugfix: inputs/.TNRS/schema.sql: derived columns: use "Accepted_family__@TNRS__@vegpath.org" instead of "*Accepted_name_family" because "*Accepted_name_family" is sometimes missing
- 05:24 AM Revision 14322: bugfix: derived_col_constraints_enable(): also need to update col comments, in addition to table__fill_derived()
- 04:54 AM Revision 14321: schemas/util.sql: check_constraint_enabled(): made it IMMUTABLE
- 04:48 AM Revision 14320: bugfix: schemas/util.sql: derived_col_constraints_enable(): need to update table__fill_derived() w/ formatting changes from enabling constraints
- 04:40 AM Revision 14319: bugfix: schemas/util.sql: derived_col_constraints_enable(): need to update table__fill_derived() w/ formatting changes from enabling constraints
- 04:12 AM Revision 14318: fix: inputs/.TNRS/schema.sql: taxon_match: added derived column "Accepted_family__@TNRS__@vegpath.org", which is needed because "*Accepted_name_family" isn't always populated
- 04:01 AM Revision 14317: bugfix: inputs/.TNRS/schema.sql: taxon_match: to add columns in the middle: also need to run util.derived_cols_repopulate() since the dependency order has changed
- 03:57 AM Revision 14316: fix: schemas/util.sql: derived_cols_populate(): documented that this function will sometimes not repopulate columns when needed, such as after changing the column order. in these cases, it is necessary to run util.derived_cols_repopulate() manually.
- 03:53 AM Revision 14315: schemas/util.sql: derived_cols_populate(): use new derived_cols_repopulate()
- 03:51 AM Revision 14314: schemas/util.sql: added derived_cols_repopulate()
- 03:25 AM Revision 14313: fix: inputs/.TNRS/schema.sql: taxon_match: COMMENT: to add columns in the middle: also need to run util.derived_cols_update()
- 03:23 AM Revision 14312: fix: inputs/.TNRS/schema.sql: taxon_match: COMMENT: updated util.derived_cols_sync() to util.derived_cols_update()
- 03:20 AM Revision 14311: inputs/.TNRS/schema.sql: reordered derived columns in dependency order
- 03:12 AM Revision 14310: inputs/.TNRS/schema.sql: reordered derived columns in dependency order
- 02:44 AM Revision 14309: bugfix: schemas/util.sql: derived_cols_trigger_update(): need to clear derived cols first so old values won't be used in calculations
- 02:36 AM Revision 14308: inputs/.TNRS/schema.sql: derived_cols_trigger_update(): put expr on same line as var, to save space
- 02:33 AM Revision 14307: inputs/.TNRS/schema.sql: derived_cols_trigger_update(): removed lines between statements, which add too much space
- 02:27 AM Revision 14306: fix: schemas/util.sql: derived_cols_trigger_update(): don't indent expr because may have multiple lines
- 02:25 AM Revision 14305: bugfix: schemas/util.sql: derived_col_update(): set_comment(): vars can't have same name as params, which will be substituted
- 01:48 AM Revision 14304: inputs/.TNRS/schema.sql: taxon_match: added derived column "[accepted_]morphospecies[_binomial]__@Brad__.TNRS@vegpath.org"
- 01:37 AM Revision 14303: inputs/.TNRS/schema.sql: taxon_match: added derived column "[parsed_]morphospecies[_suffix]__@Brad__.morphosp@vegpath.org"
- 01:27 AM Revision 14302: added web/people/Brad_Boyle/morphosp symlink to morphospecies
- 12:37 AM Revision 14301: added derived/TNRS/Morphospecies and scope.eml* from Brad
- 12:36 AM Revision 14300: added web/people/Brad_Boyle/morphospecies redirect to Morphospecies and scope.eml.txt
- 12:35 AM Revision 14299: added web/people/Brad_Boyle/Morphospecies and scope.eml.txt symlink
- 12:34 AM Revision 14298: added derived/TNRS/Morphospecies and scope.eml* from Brad
- 12:27 AM Revision 14297: added web/people/Brad_Boyle/TNRS redirect to Modifications to procedure to scrubbing names using TNRS.docx
- 12:27 AM Revision 14296: added web/people/Brad_Boyle/Modifications to procedure to scrubbing names using TNRS.docx symlink
- 12:18 AM Revision 14295: added derived/TNRS/Modifications to procedure to scrubbing names using TNRS.docx from Brad
07/21/2014
- 11:59 PM Revision 14294: schemas/public_.sql: analytical_stem: synced to analytical_stem_view
- 11:56 PM Revision 14293: bugfix: schemas/util.sql: view_body(): need to match view defs without trailing \n
- 11:53 PM Revision 14292: bugfix: schemas/util.sql: view_body(): need to support both CREATE statements and view bodies
- 11:23 PM Revision 14291: bugfix: schemas/util.sql: raise(): use eval_raw() so that functions called by eval() (such as view_def_to_orig()) can use raise() for debugging
- 09:30 PM Revision 14290: schemas/util.sql: eval(): use new eval_raw()
- 09:29 PM Revision 14289: schemas/util.sql: added eval_raw()
- 07:52 PM Revision 14288: bugfix: schemas/util.sql: view_is_automatically_updatable(), view_is_subset(): use util.view_body() so not matching COMMENT statements, etc
- 07:35 PM Revision 14287: schemas/util.sql: added view_body()
- 07:03 PM Revision 14286: bugfix: schemas/util.sql: view_is_subset(): need to support "(-x-)" in col name
- 06:34 PM Revision 14285: schemas/util.sql: added copy_struct(from_ regtype, to_ text) and copy_data(from_ regtype, to_ text), and use these in a general-purpose copy()
- 05:44 PM Revision 14284: inputs/.TNRS/schema.sql: added new derived columns to derived views
- 05:30 PM Revision 14283: fix: schemas/util.sql: derived_col_update(): column comment: formula: put on own line to support multiline exprs
- 05:26 PM Revision 14282: bugfix: schemas/util.sql: derived_col_update(): column comment: need to use expr from DB, not as provided by user, to avoid this changing when derived_col_update() is rerun
- 05:13 PM Revision 14281: inputs/.TNRS/schema.sql: taxon_match: added derived columns scrubbed_*
- 05:03 PM Revision 14280: bugfix: schemas/util.sql: derived_col_expr_from_check_constraint(): need to handle strs with \n after IS DISTINCT FROM
- 04:34 PM Revision 14279: inputs/.TNRS/schema.sql: taxon_match: put matched-taxon derived columns before accepted-taxon derived columns
- 04:25 PM Revision 14278: inputs/.TNRS/schema.sql: taxon_match: added derived column "[matched_]species[_binomial]~(-Accepted_-)__@TNRS__@vegpath.org"
- 04:08 PM Revision 14277: inputs/.TNRS/schema.sql: _matched_has_accepted: renamed to matched_has_accepted because the leading _ did not create vertical alignment anyway, due to different quoting syntaxes
- 04:05 PM Revision 14276: schemas/util.sql: added derived_col_name_from_check_constraint()
- 03:48 PM Revision 14275: schemas/util.sql: derived_col_update(): add steps to rename column
- 03:42 PM Revision 14274: web/username_prefix.php: transposes: removed recommendation for vertical alignment because different quoting syntaxes, etc. make it difficult to have a consistent alignment anyway
- 03:34 PM Revision 14273: bugfix: schemas/util.sql: derived_col_update(): don't set_comment() until CHECK constraint exists so that we can use its canon-ed formula
- 03:24 PM Revision 14272: bugfix: schemas/util.sql: check_constraint_expr_enable(): don't remove suffix without also removing prefix. this is done by using extract_by_regexp() instead. also changed check_constraint_enabled() to use regexps to match this.
- 03:08 PM Revision 14271: bugfix: schemas/util.sql: extract_by_regexp(): needs surrounding COALESCE() in case no match
- 03:05 PM Revision 14270: schemas/util.sql: added extract_by_regexp()
- 09:02 AM Revision 14269: inputs/.TNRS/schema.sql: taxon_match: added derived column _matched_has_accepted
- 08:47 AM Revision 14268: inputs/.TNRS/schema.sql: added new derived columns to derived views
- 08:44 AM Revision 14267: inputs/.TNRS/schema.sql: taxon_match: added derived columns "[matched_]scientificName[_with_author]__@DwC__@vegpath.org", "[accepted_]scientificName[_with_author]__@DwC__@vegpath.org"
- 08:29 AM Revision 14266: inputs/.TNRS/schema.sql: taxon_match: added derived columns "[accepted_]Infraspecific_rank[_abbr]__@TNRS__@vegpath.org", "[accepted_]infraspecificEpithet__@DwC__@vegpath.org"
- 08:26 AM Revision 14265: inputs/.TNRS/schema.sql: taxon_match: added derived column "__accepted_infraspecific_{rank,epithet}"
- 08:19 AM Revision 14264: inputs/.TNRS/schema.sql: taxon_match: added derived column __accepted_infraspecific_label
- 08:12 AM Revision 14263: bugfix: schemas/util.sql: canon_sql(): need to use the right search_path
- 08:12 AM Revision 14262: bugfix: schemas/util.sql: check_constraint view: def: must use our wrapper so that the right search_path is used
- 08:08 AM Revision 14261: schemas/util.sql: added pg_get_expr() wrapper, which sets search_path to pg_temp
- 07:41 AM Revision 14260: bugfix: schemas/util.sql: check_constraint_expr_*(): updated strings for new Mac Postgres update, which was installed on reboot
- 07:40 AM Revision 14259: bugfix: schemas/util.sql: check_constraint_expr_*(): updated strings for new Mac Postgres update, which was installed on reboot
- 06:55 AM Revision 14258: bugfix: schemas/util.sql: derived_col_update(): steps to modify: also need to run util.derived_cols_populate()
- 06:54 AM Revision 14257: inputs/.TNRS/schema.sql: taxon_match: added derived columns "[accepted_]genus__@DwC__@vegpath.org", "[accepted_]specificEpithet__@DwC__@vegpath.org"
- 06:50 AM Revision 14256: bugfix: schemas/util.sql: check_constraint view: can't use consrc because this doesn't get auto-updated with column renames
- 06:37 AM Revision 14255: bugfix: schemas/util.sql: derived_col_constraints_enable(): need final `SELECT NULL::void` so don't fold away functions called in previous query
- 06:26 AM Revision 14254: inputs/.TNRS/schema.sql: taxon_match: ran derived_cols_update()
- 06:19 AM Revision 14253: bugfix: derived_col_constraint_initially_enabled(): must be both already enabled *and* unchanged to be initially enabled (r14240 was only partially correct)
- 06:08 AM Revision 14252: schemas/util.sql: derived_cols_trigger_update(): to regenerate instructions: use util.derived_cols_update(), which also performs other updates
- 06:07 AM Revision 14251: schemas/util.sql: derived_cols_sync(): renamed to derived_cols_trigger_update() for clarity
- 06:05 AM Revision 14250: schemas/util.sql: added derived_cols_update()
- 05:58 AM Revision 14249: schemas/util.sql: derived_cols_sync(): renamed to derived_cols_trigger_update() for clarity
- 05:54 AM Revision 14248: schemas/util.sql: derived_col_update(): add comment describing how to modify derived column
- 05:41 AM Revision 14247: schemas/util.sql: added quote_typed(derived_col_def)
- 05:37 AM Revision 14246: schemas/util.sql: added quote_typed(col)
- 05:14 AM Revision 14245: schemas/util.sql: derived_cols_sync(): table__fill_derived(): document in the function comment how to regenerate it
- 05:08 AM Revision 14244: schemas/util.sql: added mk_set_comment(regprocedure, text)
- 05:07 AM Revision 14243: schemas/util.sql: set_comment(): accept anything with a mk_set_comment()
- 03:39 AM Revision 14242: fix: schemas/util.sql: check_constraint_expr_disable(): need to use CASE instead of `true OR ` because OR always evaluates both sides, slowing down constraint evaluation
- 03:23 AM Revision 14241: fix: schemas/util.sql: check_constraint view: conrelid: don't cast to regclass so can use index scan
- 03:21 AM Revision 14240: bugfix: derived_col_constraint_initially_enabled(): use whether existing constraint is already enabled, not whether expr is unchanged (which does not affect the *enabled* status)
- 02:52 AM Revision 14239: schemas/util.sql: check_constraint_canon_sql(): use util.debug_print_return_value() to assist debugging
- 02:49 AM Revision 14238: bugfix: schemas/util.sql: check_constraint_replace(): need to use util.check_constraint_canon_sql() instead of util.canon_sql()
- 02:40 AM Revision 14237: schemas/util.sql: added check_constraint_canon_sql(). this must be used instead of plain util.canon_sql() because unlike queries, CHECK constraints are not simplified, just standardized.
- 02:06 AM Revision 14236: schemas/util.sql: check_constraint_replace(): support not canon-ing def.expr
- 12:23 AM Revision 14235: bugfix: schemas/util.sql: CHECK constraint exprs: need to remove the enclosing () to be consistent everywhere
07/20/2014
- 11:59 PM Revision 14234: bugfix: schemas/util.sql: canon_sql(): need to include the enclosing () because the DB includes them
- 11:42 PM Revision 14233: fix: schemas/util.sql: canon_sql(): use explain()'s verbose mode to facilitate debugging
- 11:38 PM Revision 14232: bugfix: schemas/util.sql: explain(): need to restore NOTICEs in verbose mode
- 11:22 PM Revision 14231: schemas/util.sql: explain(): support running in verbose mode
- 11:19 PM Revision 14230: bugfix: schemas/util.sql: check_constraint_replace(): need to canon the expr, using new util.canon_sql(), so it can be compared with the expr in the DB
- 11:18 PM Revision 14229: schemas/util.sql: added canon_sql(expr text, table_ regclass)
- 10:45 PM Revision 14228: fix: schemas/util.sql: check_constraint view: *don't* use pretty-print mode, because this format does not match util.canon_sql()
- 10:43 PM Revision 14227: fix: schemas/util.sql: check_constraint_expr_enable(), derived_col_expr_from_check_constraint(): trim parens in order to support non-pretty-print mode, which is needed with canon_sql()
- 09:27 PM Revision 14226: schemas/util.sql: added canon_sql(expr text)
- 09:21 PM Revision 14225: schemas/util.sql: added trim_parens()
- 08:19 PM Revision 14224: schemas/util.sql: added copy_struct(from_ regtype, to_ text) and copy_data(from_ regtype, to_ text), and use these in a general-purpose copy()
- 08:02 PM Revision 14223: bugfix: schemas/util.sql: copy(from_ regtype, to_ text): must be declared STRICT to handle NULL properly
- 07:58 PM Revision 14222: schemas/util.sql: added copy(from_ regtype, to_ text)
- 07:39 PM Revision 14221: fix: schemas/util.sql: is_raise_condition(): should be called is_log_level()
- 07:36 PM Revision 14220: schemas/util.sql: raise(): support SQLSTATE values
- 07:33 PM Revision 14219: bugfix: schemas/util.sql: raise_expr(raise_type): only treat as raise condition if not SQLSTATE, to allow custom SQLSTATEs
- 07:30 PM Revision 14218: schemas/util.sql: raise(): support SQLSTATE values
- 07:28 PM Revision 14217: schemas/util.sql: added raise_expr(raise_type)
- 07:28 PM Revision 14216: schemas/util.sql: added is_raise_condition()
- 07:14 PM Revision 14215: schemas/util.sql: added is_sqlstate()
- 06:44 PM Revision 14214: schemas/util.sql: typeof(expr text, table_ regtype): use simpler util.eval2val() instead of EXECUTE
- 06:13 PM Revision 14213: fix: schemas/util.sql: derived_col_constraint_initially_enabled(): also should be initially enabled if expr unchanged, to avoid re-running constraint on all rows
- 06:09 PM Revision 14212: schemas/util.sql: derived_col_update(): use new util.derived_col_constraint_initially_enabled()
- 06:09 PM Revision 14211: schemas/util.sql: added derived_col_constraint_initially_enabled()
- 06:05 PM Revision 14210: schemas/util.sql: added derived_col_expr(col)
- 05:56 PM Revision 14209: derived_col_constraint_enable(): documented that this now avoids re-running constraint on all rows if already enabled
- 05:43 PM Revision 14208: fix: check_constraint_replace(): avoid re-running the constraint on all rows if expr hasn't changed
- 05:40 PM Revision 14207: schemas/util.sql: check_constraint_replace(): shortened param to `def` for readability
- 05:39 PM Revision 14206: schemas/util.sql: added check_constraint_expr(constraint_ table_item)
- 04:34 PM Revision 14205: schemas/util.sql: added derived_col_update()
- 04:31 PM Revision 14204: schemas/util.sql: check_constraint__disabled(): made disabled configurable (renaming func to check_constraint())
- 04:20 PM Revision 14203: schemas/util.sql: added derived_col_constraint_expr()
- 04:07 PM Revision 14202: schemas/util.sql: derived_col_constraints_enable(): use new derived_col_constraint_enable()
- 04:06 PM Revision 14201: schemas/util.sql: added derived_col_constraint_enable()
- 03:52 PM Revision 14200: schemas/util.sql: added table_has_data()
- 03:47 PM Revision 14199: schemas/util.sql: added col_add(derived_col_def)
- 03:44 PM Revision 14198: schemas/util.sql: added typeof(text, regclass)
- 03:40 PM Revision 14197: bugfix: /README.TXT: Full database import: added warnings that you should not run backups/pg_snapshot while the import is running, nor until the previous import has been replaced
- 03:22 PM Revision 14196: schemas/util.sql: added regtype(regclass)
- 02:52 PM Revision 14195: schemas/util.sql: added col_add()
- 02:32 PM Revision 14194: schemas/util.sql: added col_def type
- 07:22 AM Revision 14193: schemas/util.sql: derived_cols_populate(): only run if needed
- 07:09 AM Revision 14192: schemas/util.sql: added derived_cols_populated()
- 07:09 AM Revision 14191: schemas/util.sql: added derived_cols_populated()
- 07:08 AM Revision 14190: schemas/util.sql: added derived_col_populated()
- 07:07 AM Revision 14189: schemas/util.sql: added check_constraint_enabled()
- 06:46 AM Revision 14188: schemas/util.sql: remove_prefix(), remove_suffix(): use new starts_with(), ends_with()
- 06:45 AM Revision 14187: schemas/util.sql: added starts_with(), ends_with()
- 05:57 AM Revision 14186: schemas/util.sql: derived_cols_populate(): also run util.derived_col_constraints_enable() once populated
- 05:55 AM Revision 14185: schemas/util.sql: added derived_col_constraints_enable()
- 05:28 AM Revision 14184: schemas/util.sql: added check_constraint__disabled(), which creates a CHECK constraint which is initially disabled
- 05:27 AM Revision 14183: schemas/util.sql: added check_constraint__disabled(), which creates a CHECK constraint which is initially disabled
- 05:07 AM Revision 14182: fix: schemas/util.sql: run_triggers(): need to re-sort rows if table needs this
- 05:05 AM Revision 14181: schemas/util.sql: added try_cluster()
- 04:05 AM Revision 14180: schemas/util.sql: added derived_cols_populate()
- 04:04 AM Revision 14179: schemas/util.sql: added run_triggers(regclass)
- 04:00 AM Revision 14178: schemas/util.sql: added first_col(regclass)
- 02:42 AM Revision 14177: fix: schemas/util.sql: read-only functions should be STABLE rather than VOLATILE
- 02:42 AM Revision 14176: bugfix: schemas/util.sql: functions that access the DB must be STABLE rather than IMMUTABLE
- 02:23 AM Revision 14175: bugfix: schemas/util.sql: derived_col_defs(): derived columns *must* be returned in table order (which should = dependency order), so that they are populated in dependency order
- 02:17 AM Revision 14174: bugfix: schemas/util.sql: remove_prefix(), remove_suffix(): must be declared STRICT to handle NULL properly
- 02:16 AM Revision 14173: bugfix: schemas/util.sql: remove_suffix(): need to use quote_nullable(str) so a NULL str doesn't NULL out the entire message
- 02:10 AM Revision 14172: bugfix: schemas/util.sql: remove_prefix(): need to use quote_nullable(str) so a NULL str doesn't NULL out the entire message
- 02:09 AM Revision 14171: bugfix: schemas/util.sql: raise(): use new quote_as_str() to avoid "RAISE statement option cannot be null" errors
- 02:08 AM Revision 14170: schemas/util.sql: added quote_as_str()
07/19/2014
- 11:23 PM Revision 14169: schemas/util.sql: added CAST (util.col AS util.derived_col_def)
- 11:05 PM Revision 14168: schemas/util.sql: derived_col_def(col): use existing util.check_constraint_def()
- 11:01 PM Revision 14167: schemas/util.sql: added CAST (util.col AS util.table_item)
- 10:50 PM Revision 14166: schemas/util.sql: added derived_col_def(col)
- 10:30 PM Revision 14165: schemas/util.sql: added check_constraint_disable()
- 10:23 PM Revision 14164: bugfix: schemas/util.sql: check_constraint_drop(): need IF EXISTS
- 10:09 PM Revision 14163: schemas/util.sql: added check_constraint_expr_disable()
- 09:50 PM Revision 14162: schemas/util.sql: check_constraint_enable(): documented that this is idempotent
- 09:21 PM Revision 14161: inputs/.TNRS/schema.sql: taxon_match: documented that whenever columns are renamed, util.derived_cols_sync() should be run
- 09:14 PM Revision 14160: schemas/util.sql: derived_cols_sync(): also create trigger, since this is necessary for a complete sync
- 09:13 PM Revision 14159: schemas/util.sql: added name(regprocedure)
- 09:02 PM Revision 14158: schemas/util.sql: derived_cols_sync(): set the trigger name in a var
- 08:52 PM Revision 14157: schemas/util.sql: added check_constraint_enable()
- 08:46 PM Revision 14156: schemas/util.sql: added check_constraint_replace()
- 08:43 PM Revision 14155: schemas/util.sql: added CAST (util.table_item AS util.check_constraint_def)
- 08:38 PM Revision 14154: schemas/util.sql: added check_constraint_drop()
- 08:02 PM Revision 14153: schemas/util.sql: derived_col_expr_from_check_constraint(): enable initially-disabled exprs
- 08:00 PM Revision 14152: schemas/util.sql: added check_constraint_expr_enable()
- 07:42 PM Revision 14151: fix: schemas/util.sql: check_constraint view: def: use pg_get_expr() with pretty-print, to avoid extra () that complicate parsing
- 06:54 PM Revision 14150: inputs/.TNRS/schema.sql: taxon_match__fill_derived(): updated using new util.derived_cols_sync()
- 06:50 PM Revision 14149: schemas/util.sql: added derived_cols_sync()
- 06:45 PM Revision 14148: bugfix: CAST (util.check_constraint_def AS util.derived_col_def): need to extract the derived column expr *from* the CHECK constraint expr
- 06:44 PM Revision 14147: schemas/util.sql: added derived_col_expr_from_check_constraint()
- 06:39 PM Revision 14146: schemas/util.sql: added extract_str()
- 06:34 PM Revision 14145: schemas/util.sql: added remove_suffix()
- 06:21 PM Revision 14144: schemas/util.sql: added remove_prefix()
- 05:47 PM Revision 14143: schemas/util.sql: added CAST (util.table_item AS util.col)
- 05:37 PM Revision 14142: schemas/util.sql: derived_col_defs(): return derived_col_def instead of check_constraint_def
- 05:36 PM Revision 14141: schemas/util.sql: added derived_col_def type with cast from util.check_constraint_def
- 04:51 PM Revision 14140: schemas/util.sql: added derived_col_defs()
- 04:35 PM Revision 14139: schemas/util.sql: check_constraint_defs(): use new util.check_constraint view
- 04:30 PM Revision 14138: schemas/util.sql: added check_constraint view
- 03:53 PM Revision 14137: schemas/util.sql: added check_constraint_defs()
- 03:53 PM Revision 14136: schemas/util.sql: added check_constraint_def type
- 03:51 PM Revision 14135: schemas/util.sql: added table_item type
- 01:59 PM Revision 14134: schemas/util.sql: col_ref: renamed to just col
- 01:38 PM Revision 14133: web/username_prefix.php: remove padding used to visually separate elements: documented that this leaves embedded __ (eg. x__y) untouched
- 02:52 AM Revision 14132: fix: inputs/.TNRS/schema.sql: taxon_match: to populate a new column: also need to recluster table so rows are exported in sorted order
- 02:47 AM Revision 14131: schemas/util.sql: cluster(): support omitting index to recluster a table
- 02:26 AM Revision 14130: schemas/util.sql: cluster_once(): use util.cluster()
- 02:24 AM Revision 14129: schemas/util.sql: cluster_once(): documented that this runs cluster only *once* if called repeatedly
- 02:24 AM Revision 14128: schemas/util.sql: added cluster(), which reclusters each time
- 02:10 AM Revision 14127: schemas/util.sql: cluster_once(): use simpler `LANGUAGE sql` now that we're using util.eval() instead of EXECUTE
- 02:03 AM Revision 14126: schemas/util.sql: cluster_once(): use util.eval() instead of EXECUTE to print the CLUSTER statement (if any) it's using
07/18/2014
- 11:40 PM Revision 14125: fix: inputs/.TNRS/schema.sql: taxon_match: cluster the table so the rows are always exported in the same order, even when an UPDATE statement is used to run triggers
- 11:28 PM Revision 14124: bugfix: schemas/util.sql: cluster_once(): need to use just the name of the index, without the schema prefix
- 05:00 PM Revision 14123: schemas/util.sql: cluster_once(): use simpler IS DISTINCT FROM instead of array !=
- 03:18 PM Revision 14122: inputs/.TNRS/schema.sql: added __accepted_{genus,specific_epithet} derived column, using the steps at wiki.vegpath.org/Postgres_queries#maintaining-derived-column-formulas-across-column-renames
- 02:01 AM Revision 14121: web/username_prefix.php: remove padding used to visually separate elements: documented that this also removes a leading __ that indicates an internal field
07/17/2014
- 07:18 PM Revision 14120: web/links/index.htm: updated to Firefox bookmarks: Quick links: added VegBIEN data dictionary spreadsheet, TNRS data dictionary.
- 06:15 PM Revision 14119: fix: web/username_prefix.php: transposes: changed recommended syntax to _y~x@url so that _y~ lines up vertically with [y_] paths
- 06:11 PM Revision 14118: web/username_prefix.php: removed special category comments syntax because this is now handled by transposes and insertion comments
- 06:10 PM Revision 14117: web/username_prefix.php: added support for transposes ( y~x_@url -> x_y@url , Y~x@url -> xY@url ), which can be used to put the category in a column name first without duplicating the category
- 05:35 PM Revision 14116: web/username_prefix.php: removed linewraps code because Google spreadsheets no longer supports wrapping on a - , and because a shorter syntax for category comments is now available
- 05:32 PM Revision 14115: web/username_prefix.php: category comments: moved _-_ outside [] in comparison example since it will be deleted separately
- 05:27 PM Revision 14114: bugfix: web/username_prefix.php: category comments: need /__/ around regexp
- 05:24 PM Revision 14113: web/username_prefix.php: category comments: added support for a shorter syntax than [x_-_]y@url : x+y@url -> y@url
- 04:14 PM Revision 14112: schemas/vegbien.ERD.mwb: updated
- 04:13 PM Revision 14111: fix: inputs/.TNRS/schema.sql: taxon_match: added back * prefixes on TNRS-namespace column names
- 03:34 PM Revision 14110: inputs/.TNRS/schema.sql: MatchedTaxon: taxon_best_match: don't alias to `s` since this is no longer a nested select
- 03:32 PM Revision 14109: inputs/.TNRS/schema.sql: MatchedTaxon: nested select: use just taxon_best_match since this no longer performs renamings
- 03:27 PM Revision 14108: inputs/.TNRS/schema.sql: MatchedTaxon: renamed output columns to match input columns, so that the nested select would not be performing any renamings
- 03:05 PM Revision 14107: fix: inputs/.TNRS/schema.sql: MatchedTaxon: added all columns from taxon_match (some of them apparently hadn't been included in MatchedTaxon)
- 03:01 PM Revision 14106: fix: inputs/.TNRS/schema.sql: reordered columns to match taxon_match
- 02:38 PM Revision 14105: fix: inputs/.TNRS/schema.sql: MatchedTaxon: added all columns from taxon_match (some of them apparently hadn't been included in MatchedTaxon)
- 01:02 PM Revision 14104: fix: inputs/.TNRS/schema.sql: added back Family_matched, which had gotten merged with Name_matched_accepted_family when TNRS temporarily stopped populating Name_matched_accepted_family
- 12:32 PM Revision 14103: inputs/.TNRS/schema.sql: MatchedTaxon: taxonomicStatus: moved outside the inner SELECT so that the inner SELECT would consist solely of renamings
- 12:12 PM Revision 14102: inputs/.TNRS/schema.sql: MatchedTaxon: removed unneeded "Name_matched." prefix on source-specific names (only the * is needed because there is only one table)
- 12:10 PM Revision 14101: bugfix: schemas/util.sql: view_is_subset(): auto-updatable views that are *not* plain subsets: include ones with WHERE clauses
- 12:03 PM Revision 14100: bugfix: schemas/util.sql: view_is_subset(): also need to handle auto-updatable views that are *not* plain subsets (there are some of these)
- 09:05 AM Revision 14099: bin/import_all: hidden_srcs(): removed `by_col=1` because these should be done in the same mode as the main datasources
07/16/2014
- 07:55 PM Revision 14098: schemas/public_.sql: removed no longer used threatened_taxonlabel. use iucn_red_list instead.
- 07:39 PM Revision 14097: fix: schemas/public_.sql: analytical_stem_view: don't use threatened_taxonlabel, which was never populated correctly. instead, this can eventually be rewritten to use new iucn_red_list.
- 07:13 PM Revision 14096: bin/make_analytical_db: removed threatened_taxonlabel because this is now handled by iucn_red_list
- 07:12 PM Revision 14095: bin/make_analytical_db: added iucn_red_list_view_modify()
- 06:48 PM Revision 14094: bin/make_analytical_db: removed unused code to create views in the analytical_db schema
- 06:35 PM Revision 14093: bin/make_analytical_db: merged mk_table and mk_analytical_table since they now do the same thing
- 06:34 PM Revision 14092: bin/make_analytical_db: removed unused code to create views in the analytical_db schema
- 05:05 PM Revision 14091: /README.TXT: Notes on system stability: added warning that when shutting down the VM, one should always first stop Postgres, to prevent the OS from SIGKILLing it
- 05:04 PM Revision 14090: inputs/SALVIAS/Source/header.csv: updated
- 03:50 PM Revision 14089: bugfix: bin/with_all: isset(): need to use `&>/dev/null` instead of `>&-`, etc because closing an fd causes declare to return false
- 03:31 PM Revision 14088: bugfix: bin/with_all, import_all: don't disown processes because they should be auto-killed if the shell is (disown was only needed before we used screen)
- 03:24 PM Revision 14087: bugfix: Full database import: need nested shell to prevent errexit from closing the window
- 01:24 PM Revision 14086: bugfix: schemas/public_.sql: analytical_stem_view_modify(): added columns: need to include COMMENT statements
- 01:21 PM Revision 14085: bugfix: schemas/util.sql: comment(element oid): need `objsubid = 0` filter so this doesn't use comments for any other objsubids
- 12:53 PM Revision 14084: schemas/public_.sql: added iucn_red_list_view_modify() and use it in iucn_red_list_view's "after updating this" instructions
- 12:47 PM Revision 14083: bugfix: schemas/util.sql: rematerialize_view(): also need to handle the target table's dependent views
- 12:38 PM Revision 14082: bugfix: schemas/util.sql: mk_drop_from_create(): need to use `IF EXISTS` so that the resulting statement also works if the target object does not yet exist
- 12:24 PM Revision 14081: schemas/util.sql: mk_drop_from_create(): also support `SELECT util.drop_*()` statements, in addition to DROP
- 11:37 AM Revision 14080: inputs/.TNRS/Source/test.xml.ref: updated
07/15/2014
- 11:14 PM Revision 14079: bugfix: /README.TXT: logs glob: don't unintentionally match . or ..
- 10:43 PM Revision 14078: inputs/VegBank/stemlocation_/test.xml.ref: updated
- 10:41 PM Revision 14077: /README.TXT: Full database import: `make test by_col=1`: updated runtime (1 h)
- 10:07 PM Revision 14076: inputs/TEX/Specimen2/test.xml.ref: updated
- 09:35 PM Revision 14075: inputs/GBIF/raw_occurrence_record_plants/test.xml.ref: updated
- 05:50 PM Task #887 (Rejected): fix disk space leak that fills the disk and crashes the import
- 05:48 PM Task #902 (Resolved): fix bug that causes joining on the wrong columns in the import
- 05:47 PM Task #905 (Resolved): narrow down the cause of the import bug (incorrect join columns and disk space leak)
- bug fixed in r14074
- 05:44 PM Revision 14074: bugfix: lib/sql_io.py: put_table(): handle_MissingCastException(): when updating join_cols, don't add *new* entry for join_cols[out_col], only update existing one. this fixes #902 (import bug), and with #902 fixed, #887 (disk space leak) should no longer occur.
- 04:49 PM Revision 14073: bin/import_all: delete_logs(): documented that `trap EXIT` doesn't run until *shell* exit
- 04:48 PM Revision 14072: bin/import_all: delete_logs(): print when this happens, so it can be verified that it's happening properly
- 04:32 PM Revision 14071: bugfix: bin/import_all: need to run delete_logs manually because `trap EXIT` doesn't run until bg cmds done
- 04:28 PM Revision 14070: bin/import_all: delete_logs: moved testing of whether to delete logs to delete_logs() so that delete_logs() can be run regardless of the $delete_logs setting
- 03:58 PM Revision 14069: bugfix: bin/import_all: delete_logs(): also need to match log filenames when n=""
- 03:11 PM Revision 14068: bugfix: /Makefile: postgres_restart-*: argument to echo always needs quotes
- 03:05 PM Revision 14067: bugfix: /Makefile: $(pg_ctl-*): tell user that if it doesn't work, need to run `rm $(macPostgresDir)/postmaster.pid` and retry
- 02:57 PM Revision 14066: /Makefile: $(pg_ctl-Darwin): documented the pg_ctl version of this command in case `brew services` doesn't work
- 02:39 PM Revision 14065: bugfix: bin/with_all: isset(): need to use `>&- 2>&-` because &> does not work with - as the dest
- 01:19 PM Revision 14064: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: determine if the revision works: use the in# of the query instead of waiting an arbitrary time, because this is more accurate in determining whether the import is successful
- 12:52 PM Revision 14063: config/VirtualBox_VMs/vegbiendev/README.TXT: regressions that occurred during the revision range: named these "concurrent regressions"
- 12:45 PM Revision 14062: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: need to update system time so wait times can be calculated correctly
- 12:22 PM Revision 14061: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: documented that the patches are specifically for regressions that occurred during the revision range, not all bugs that were fixed in that range
- 12:14 PM Revision 14060: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: waiting for query: increased time to min. 10 min because some revisions take longer
- 12:09 PM Revision 14059: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bin/import_all: added step to type y and press ENTER
- 12:07 PM Revision 14058: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to determine if the revision works, and then stop the import
- 11:27 AM Revision 14057: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12994
- 11:09 AM Revision 14056: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: use for loop to avoid needing to repeat the command
- 11:05 AM Revision 14055: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12760
- 10:54 AM Revision 14054: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: --fuzz apparently needs to be 3 even though we have only 2 lines of context
- 10:52 AM Revision 14053: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: use --fuzz to allow patching even when surrounding lines have changed
- 10:44 AM Revision 14052: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: sorted by revision #
- 10:43 AM Revision 14051: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12865
- 10:33 AM Revision 14050: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: patch: need to use --force to avoid prompting user
- 10:16 AM Revision 14049: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: `svn revert`: need to use `--depth infinity` when reverting more than a file
- 10:14 AM Revision 14048: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: use `patch` instead of `svn merge` because `svn merge` sometimes inexplicably doens't apply patches
- 09:43 AM Revision 14047: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: `svn revert` all changes instead of specific files, to support bugfixes to other files
- 09:42 AM Revision 14046: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: lib/sh/util.sh bugfix: use `svn merge` instead of `sed --in-place` because this can be used for more complex changes
- 09:41 AM Revision 14045: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: lib/sh/util.sh bugfix: documented revision
- 09:20 AM Revision 14044: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: need to apply r12872 bugfix to lib/sh/util.sh because it occurred after r12025
- 09:10 AM Revision 14043: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make` commands: use `yes|` to avoid needing to confirm prompts
- 09:08 AM Revision 14042: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make mk_db ...`: use `yes|` to avoid needing to confirm prompts
- 08:56 AM Revision 14041: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: to test a different revision: added step that if any of the commands after `svn up` are broken, try a different revision
- 08:46 AM Revision 14040: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make db`: need to run `schemas/reinstall` manually because older versions of the codebase just used `schemas/install`
07/14/2014
- 11:30 PM Revision 14039: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: documented the range from within the revision should be chosen
- 09:19 PM Revision 14038: fix: inputs/.IRMNG/*/: added _no_import because this datasource is only used as a lookup table and does not need to be imported/scrubbed
- 09:18 PM Revision 14037: config/VirtualBox_VMs/vegbiendev/README.TXT: `svn up`: documented runtime (1 min)
- 08:24 PM Revision 14036: bugfix: added nodes/_no_import, because the NCBI import doesn't work due to a Postgres 9.3 bug (#859). _no_import used to not be needed because we didn't import the hidden sources.
- 08:05 PM Revision 14035: fix: bin/with_all: removed debug statements
- 05:19 PM Revision 14034: bugfix: bin/with_all: testing if @inputs is set: `"${inputs+isset}"` syntax doesn't work for empty arrays, so need to use `declare -p` instead
- 04:46 PM Revision 14033: bugfix: to set up the VM: to use a database backup: also need to `make rm_initial_public` because initdb creates a non-empty DB
- 04:42 PM Revision 14032: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to test a different revision
- 04:40 PM Revision 14031: /Makefile: db: use reinstall instead of install to allow running this on an existing DB
- 02:49 PM Revision 14030: config/VirtualBox_VMs/vegbiendev/README.TXT: `sudo apt-get dist-upgrade`: documented runtime (12 min)
- 02:47 PM Revision 14029: config/VirtualBox_VMs/vegbiendev/README.TXT: updated VM snapshot names
- 02:44 PM Revision 14028: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to install system updates: added step to restart, because some system updates require this
- 02:34 PM Revision 14027: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to install system updates: added step to restart, because some system updates require this
- 02:29 PM Revision 14026: config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to install system updates
- 01:23 PM Revision 14025: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bin/import_all: documented runtime (50 min)
- 01:07 PM Revision 14024: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to run the import
- 12:24 PM Revision 14023: config/VirtualBox_VMs/vegbiendev/README.TXT: to set up the VM: added steps to move vegbiendev.vdi to the internal hard drive
- 12:19 PM Revision 14022: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to start the VM
- 12:16 PM Revision 14021: config/VirtualBox_VMs/vegbiendev/README.TXT: moved "set up the VM" steps to separate section
- 12:06 PM Revision 14020: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: added steps to create a booted snapshot
- 12:01 PM Revision 14019: config/VirtualBox_VMs/vegbiendev/README.TXT: moved creation of pg_restore snapshot to be part of just the database backup section
- 11:52 AM Revision 14018: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: documented runtime (15 min)
- 11:51 AM Revision 14017: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: time the operation
- 11:32 AM Revision 14016: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: added name for VM snapshot after this step
- 11:31 AM Revision 14015: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: need to run as DB superuser to restore py_util
- 11:24 AM Revision 14014: config/VirtualBox_VMs/vegbiendev/README.TXT: rsync: updated runtime (25 min)
- 11:24 AM Revision 14013: config/VirtualBox_VMs/vegbiendev/README.TXT: added steps to create snapshot before running pg_restore
- 10:55 AM Revision 14012: config/VirtualBox_VMs/vegbiendev/README.TXT: added steps to configure the VM before booting
- 10:53 AM Revision 14011: config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions for selecting the initial snapshot and booting the VM
- 10:36 AM Revision 14010: config/VirtualBox_VMs/vegbiendev/README.TXT: perform long-running operations inside `screen`
- 10:35 AM Revision 14009: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to use a database backup: only run mk_db, not db, to avoid restoring schemas that are part of the backup
- 09:33 AM Revision 14008: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: schemas must be in dependency order
- 09:21 AM Revision 14007: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: don't restore analytical_db schema because we don't use it and it depends on public, which we're not restoring
- 09:16 AM Revision 14006: web/links/index.htm: updated to Firefox bookmarks: WebEx: added steps to join a test meeting. pg_restore: documented that you need to *create* the schema before you restore into it, or else you will get a cryptic error message that it's using pg_catalog instead. added Homebrew links, including instructions for setting psycopg2 up with it. added PostGIS links. username URLs: documented how to enable these in Internet Explorer. TWiki: documented what to do when moving the TWiki install dir.
- 09:06 AM Revision 14005: bugfix: lib/Firefox_bookmarks.reformat.csv: updated for new Firefox bookmarks format, which indents the <DD> tag
- 08:36 AM Revision 14004: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: need to *create* the schema before you restore into it, to avoid cryptic error messages
- 08:33 AM Revision 14003: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: schemas/postgresql.conf: need to decrease shared_buffers to match VM's allocated memory (1 GB)
- 08:18 AM Revision 14002: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: schemas/postgresql.conf: need to decrease shared_buffers to match VM's allocated memory (1 GB)
07/13/2014
- 11:50 PM Revision 14001: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: need to run `make db` to restore users, which are not part of the backup
- 11:40 PM Revision 14000: config/VirtualBox_VMs/vegbiendev/README.TXT: rsync: updated runtime (35 min)
07/11/2014
- 04:54 PM Revision 13999: config/VirtualBox_VMs/vegbiendev/README.TXT: turning off of non-ARIZ inputs: removed metadata datasources because these have been renamed to start with ".", and so aren't included in */
- 04:48 PM Revision 13998: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: pg_restore: need to use for loop because pg_restore doesn't support multiple --schema options (it just uses the last one)
- 04:35 PM Revision 13997: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added step to restore the necessary schemas from the DB
- 04:32 PM Revision 13996: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: rsync: documented runtime (13 min)
- 04:11 PM Revision 13995: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added step to fetch the database backup via rsync
- 04:04 PM Revision 13994: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: moved steps to enable the user to access VirtualBox shared folders to separate section because VirtualBox shared folders currently don't work, likely because of a bug in VirtualBox Guest Additions when installed on something other than a clean Linux install
- 03:54 PM Revision 13993: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added steps to enable the user to access VirtualBox shared folders
- 03:06 PM Revision 13992: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added section for database backup approach, as alternative to clean database approach
- 10:32 AM Revision 13991: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: use ".." instead of - to indicate a range, to avoid this being confused for a minus sign
- 10:26 AM Revision 13990: bugfix: /Makefile: postgres_restart-Darwin: need to have the user run the command outside of make because it doesn't appear to do anything when run inside make (this is the same problem as on Linux)
- 10:25 AM Revision 13989: /Makefile: $(pg_ctl-Darwin): use simpler `brew services` instead of `launchctl __/homebrew.mxcl.postgresql.plist`
- 10:03 AM Revision 13988: bugfix: bin/stop_imports: also need to include `bin/after_import`
- 09:56 AM Revision 13987: inputs/.TNRS/Source/map.csv: datasetURL: updated to indicate that this uses the TNRS dev server (the actual URL is private)
- 09:46 AM Revision 13986: fix: inputs/VASCAN/Source/map.csv: observationGranularity should not be specimen because this datasource contains only taxa
- 09:13 AM Revision 13985: bugfix: bin/import_all: now that always using log files to fix output clutter, need to delete created logs if logging is turned off
- 08:45 AM Revision 13984: bugfix: bin/import_all: don't errexit if a background process is Ctrl-C'd
- 08:41 AM Revision 13983: bugfix: bin/import_all: was run without initial "." test: don't exit nonzero because this will close the subshell
- 08:38 AM Revision 13982: bugfix: bin/import_all: ensure that this is run in a subshell, which is needed so errexits don't close the terminal window
- 08:32 AM Revision 13981: bin/import_all: documented that this must be run in a subshell (obtained by running `$0`)
- 08:25 AM Revision 13980: bugfix: bin/import_all: need to always use log files for background processes
- 08:12 AM Revision 13979: fix: bin/import_all: Source/import: don't use by_col=1 for this because it's slower for small #s of rows. by_col mode is no longer needed for metadata-only tables because these tables now have a single empty row so that they also work in row-based mode.
- 08:06 AM Revision 13978: fix: bin/import_all: hidden srcs: use with_all for this to avoid needing to list every source, and to display the backgrounded command with the variables substituted
- 07:40 AM Revision 13977: bin/import_all: TNRS, geoscrub: integrated into the list of metadata sources
- 07:39 AM Revision 13976: bin/import_all: TNRS, geoscrub: use import rather than publish because the non-imported tables have now been excluded
- 07:34 AM Revision 13975: fix: inputs/.geoscrub/geoscrub_output/: added _no_import because these tables are metadata that is used in the analytical DB. this is better than relying on bin/import_all not to import these.
07/10/2014
- 07:25 PM Revision 13974: fix: bin/import_all: updated for new metadata datasource names (see issue #940)
- 07:25 PM Revision 13973: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: updated for new metadata datasource names (see issue #940)
- 06:41 PM Task #940 (Resolved): resequence datasources in dependency order
- 04:22 PM Task #940 (Resolved): resequence datasources in dependency order
- * this enables bootstrapping the DB from a VM backup
steps
# put all [[VegBIEN_contents#datasources|metadata da... - 06:32 PM Revision 13972: fix: inputs/IUCN/: renamed to inputs/.IUCN/ (see issue #940)
- 06:04 PM Revision 13971: fix: inputs/newWorld/: renamed to inputs/.newWorld/ (see issue #940)
- 04:59 PM Revision 13970: bugfix: inputs/IRMNG/: renamed to inputs/.IRMNG/ so that this comes before TNRS, which depends on it (this is a metadata datasource, so it can start with "."). part of issue #940.
- 03:52 PM Revision 13969: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: set up the VM: added steps to create a VM snapshot when done
- 03:37 PM Revision 13968: config/VirtualBox_VMs/vegbiendev/README.TXT: `make`: documented runtime (15 min)
- 03:33 PM Revision 13967: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: removed non-applicable step about the MySQL root password, which is not used for this procedure
- 03:32 PM Revision 13966: added config/VirtualBox_VMs/vegbiendev/README.TXT with instructions to test the import
- 12:17 PM Revision 13965: inputs/GBIF/_MySQL/.rsync_ignore: don't exclude GBIFPortalDB-*.data.sql.gz, even though this is an intermediate file, because it's better to have a backup of it locally. this was excluded in r13316 (2014-4-24) to free up disk space on the local machine.
- 10:57 AM Revision 13964: fix: inputs/Madidi/LocationObservation/postprocess.sql: parse Spanish dates
- 09:45 AM Revision 13963: validation/completeness/BIENDatesPopulated.xlsx: updated: Martha added Priority column
- 08:41 AM Revision 13962: schemas/util.sql: replace_words(): use map_words() to enable inlining the concatenated hstore, so that it doesn't need to be concatenated on *each* function call
- 08:39 AM Revision 13961: schemas/util.sql: added map_words()
- 08:16 AM Revision 13960: schemas/util.sql: added date_from_spanish()
- 08:13 AM Revision 13959: schemas/util.sql: added replace_words()
- 08:09 AM Revision 13958: schemas/util.sql: added words(text)
- 08:02 AM Revision 13957: schemas/util.sql: spanish_date_words(): switched to using hstore because this is better for the algorithm that will translate these
- 07:20 AM Revision 13956: schemas/util.sql: added spanish_date_words()
- 07:19 AM Revision 13955: schemas/util.sql: added `replacement` type
- 04:09 AM Revision 13954: bugfix: mk_set_relation_metadata(): need to include col comments as well so that these are re-created properly by recreate()
- 04:07 AM Revision 13953: schemas/util.sql: added mk_set_comments(regclass)
- 04:06 AM Revision 13952: schemas/util.sql: show_set_comment(regclass): renamed to mk_set_comment() to match other mk_set_comment() functions
- 04:01 AM Revision 13951: schemas/util.sql: added mk_set_col_comments(regclass)
- 03:58 AM Revision 13950: schemas/util.sql: added cols(regclass)
- 03:56 AM Revision 13949: schemas/util.sql: added mk_set_comment(col_ref)
- 03:18 AM Revision 13948: added validation/completeness/BIENDatesPopulated.xlsx from Google spreadsheets
07/09/2014
- 02:50 PM Revision 13947: fix: mappings/VegCore-VegBIEN.csv: mapped DwC year/month/day, which are split-date alternatives to eventDate. this fixes the missing eventDate in FIA.
- 02:45 PM Revision 13946: bugfix: inputs/FIA/*/VegBIEN.csv: regenerated, replacing these with symlinks to the file used by new-style import
- 02:34 PM Revision 13945: inputs/FIA/taxon_observation.**/test.xml.ref: updated
- 02:34 PM Revision 13944: inputs/FIA/TREE/test.xml.ref: updated
- 02:29 PM Revision 13943: inputs/FIA/REF_RESEARCH_STATION/test.xml.ref: updated
- 02:13 PM Revision 13942: bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new TWiki/ location
- 02:12 PM Revision 13941: bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new VegBIEN/ location
- 12:14 PM Revision 13940: fix: inputs/bien2_traits/TraitObservation/map.csv: mapped VisitingDate to eventDate
- 12:05 PM Revision 13939: fix: inputs/Madidi/LocationObservation/postprocess.sql: populated missing eventDate from PlotInventoryName (authorEventCode)
- 09:40 AM Revision 13938: schemas/util.sql: mk_set_comment(table_ regclass, comment text): use new util.mk_set_comment(text, text)
- 09:38 AM Revision 13937: schemas/util.sql: added mk_set_comment(col col_ref, comment text)
- 09:37 AM Revision 13936: schemas/util.sql: added sql(col_ref)
- 09:35 AM Revision 13935: schemas/util.sql: added mk_set_comment(on_ text, comment text)
- 07:53 AM Revision 13934: schemas/util.sql: added comment(col_ref)
- 07:50 AM Revision 13933: schemas/util.sql: added col_num(col_ref)
- 07:03 AM Revision 13932: removed no longer needed inputs/.TNRS/grants.sql, since the grants in schema.sql are now being run
- 07:03 AM Revision 13931: bugfix: inputs/input.Makefile: sql/install: schema.sql should not be passed through pg_dump_limit because it contains GRANT statements that need to be run
- 06:22 AM Revision 13930: schemas/public_.sql: iucn_red_list_view: documented how to regenerate iucn_red_list from this
07/08/2014
- 04:17 PM Revision 13929: exports/2014-6-4.Iara_Lacher.reserve_prioritization.csv.run: documented runtime (7.5 min) and rowcount (3.5 million)
- 03:23 PM Revision 13928: bugfix: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: fixed slowdown in materializing the view, which was caused by source__observation_type(), by replacing this with a LEFT JOIN as was done for is_threatened_iucn
07/07/2014
- 07:33 AM Revision 13927: schemas/util.sql: in_south_america(): optimized by using BETWEEN instead of util.contained_within__no_dateline(). this reduces the filter time for 2014-6-4.Iara_Lacher.reserve_prioritization by about 10%.
- 07:26 AM Revision 13926: fix: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: is_threatened_iucn: generate this much faster by doing a LEFT JOIN, which can reuse the same in-memory hash index for every row
- 07:04 AM Revision 13925: bugfix: schemas/public_.sql: wrapper functions can't use util-schema types because these will cause the wrapper function to be cascadingly dropped when the util schema is reinstalled
- 06:55 AM Revision 13924: bugfix: in_south_america(): must use util.contained_within__no_dateline() instead of util.contained_within_approx() to ensure that the more accurate geometry logic is used
- 06:51 AM Revision 13923: bugfix: schemas/public_.sql: source__observation_type(): don't display NOTICEs about the search_path, because this function will be called millions of times
- 06:48 AM Revision 13922: bugfix: schemas/public_.sql: in_iucn_red_list(): don't display NOTICEs about the search_path, because this function will be called millions of times
- 06:43 AM Revision 13921: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added is_threatened_iucn
- 06:37 AM Revision 13920: schemas/public_.sql: added in_iucn_red_list()
- 06:36 AM Revision 13919: schemas/public_.sql: added iucn_red_list, materialized from iucn_red_list_view
- 06:22 AM Revision 13918: bugfix: schemas/vegbien.sql: iucn_red_list_view: need to include only names with an accepted name, and distinctify on the accepted names
- 06:17 AM Revision 13917: schemas/public_.sql: added iucn_red_list
- 06:14 AM Revision 13916: schemas/util.sql: materialize_view(): use util.copy() instead of util.materialize_query() so that all view metadata is transferred
- 06:02 AM Revision 13915: schemas/public_.sql: iucn_red_list: renamed to iucn_red_list_view since this will be materialized
- 06:01 AM Revision 13914: schemas/public_.sql: added iucn_red_list
- 05:44 AM Revision 13913: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added collection_type (using source__observation_type())
- 05:36 AM Revision 13912: schemas/public_.sql: added source__observation_type()
- 05:04 AM Revision 13911: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added accepted_subspecies, and removed accepted_taxon_name_with_author which had been substituting for it
- 04:50 AM Revision 13910: schemas/public_.sql: added subspecies() wrapper
- 04:38 AM Revision 13909: schemas/util.sql: added subspecies()
- 04:10 AM Revision 13908: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: include only coordinates in South America
- 03:55 AM Revision 13907: schemas/util.sql: added south_america(), in_south_america()
- 03:37 AM Revision 13906: fix: schemas/util.sql: OPERATOR ~@(geocoord, postgis.geometry): renamed to @ because this is not an approximate comparison for geometry
- 03:35 AM Revision 13905: fix: schemas/util.sql: contained_within_approx(geocoord, postgis.geometry): renamed to contained_within__no_dateline(__) because this is not an approximate comparison for geometry
- 03:30 AM Revision 13904: schemas/util.sql: lat_long_in_new_world(): renamed to just in_new_world() because the lat/long is implied by the param type
- 03:08 AM Revision 13903: schemas/util.sql: lat_long_in_new_world(): take a geocoord param instead of separate lat/long params
- 03:00 AM Revision 13902: backups/TNRS.backup.md5: updated
07/06/2014
- 11:08 PM Revision 13901: schemas/util.sql: added contained_within_approx(geocoord, geometry) and corresponding OPERATOR ~@(geocoord, geometry)
- 10:39 PM Revision 13900: schemas/util.sql: added OPERATOR ~@(geocoord, geography)
- 10:23 PM Revision 13899: schemas/util.sql: lat_long_in_new_world(): use new contained_within_approx(geocoord, geography)
- 10:17 PM Revision 13898: schemas/util.sql: added contained_within_approx(geocoord, postgis.geography), which enables specifying just `(lat, long)` without the ::util.geocoord type specifier
- 04:04 PM Revision 13897: schemas/util.sql: OPERATOR @(postgis.geography, postgis.geography): renamed to ~@ because it's approximate
- 03:52 PM Revision 13896: schemas/util.sql: contained_within(): renamed to contained_within_approx() because the latitude lines of geography type bounding boxes bulge outward, creating false positives above and below the bounding box
- 03:50 PM Revision 13895: schemas/util.sql: added contained_within__no_dateline(geometry, geometry) and corresponding operator @
- 02:51 PM Revision 13894: schemas/util.sql: contained_within(): renamed to contained_within_approx() because the latitude lines of geography type bounding boxes bulge outward, creating false positives above and below the bounding box
- 05:49 AM Revision 13893: schemas/util.sql: geometry(geocoord): documented that it is not possible to create a cast for this, as a bug in pg_dump prevents the cast from being exported, even when no export filters are applied
07/05/2014
- 12:40 PM Revision 13892: schemas/util.sql: point(geocoord): renamed to geometry(geocoord) since this is now a cast
- 12:03 PM Revision 13891: schemas/util.sql: point(): return geometry instead of geography to support using points with geometry arithmetic
- 11:24 AM Revision 13890: schemas/util.sql: point(): take a single util.geocoord param instead of separate lat/long
- 10:45 AM Revision 13889: schemas/util.sql: added geocoord type
07/04/2014
- 08:29 PM Revision 13888: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: include only *georeferenced* occurrences (lat/long NOT NULL)
- 08:25 PM Revision 13887: schemas/util.sql: bounding_box(): use bounding_box__no_dateline() to construct the postgis.geometry object
- 08:23 PM Revision 13886: schemas/util.sql: added bounding_box__no_dateline(), which is more accurate than util.bounding_box() (latitude lines will be straight), but geocoordinate wraparound is not supported
- 08:14 PM Revision 13885: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added functional traits that we have 1st-class columns for (dbh_cm, height_m)
- 12:29 PM Revision 13884: added exports/2014-6-4.Iara_Lacher.reserve_prioritization.csv.run
- 11:12 AM Revision 13883: inputs/publishable datasources.xlsx: updated
- 08:45 AM Revision 13882: inputs/publishable datasources.xlsx: updated
07/03/2014
- 08:13 PM Revision 13881: inputs/.TNRS/schema.sql: matchedFamily: just use Name_matched_accepted_family, because TNRS has now been reloaded so that the names that were missing this have it populated
- 09:59 AM Revision 13880: planning/meetings/BIEN conference call availability.xlsx: updated
- 03:47 AM Task #917: TNRS: Instructions for new version with TPL
- a single "-" creates strikethrough formatting, so you have to use "--" instead
- 12:34 AM Revision 13879: inputs/.TNRS/schema.sql: taxon_match: taxon_match__valid_match: replaced with taxon_best_match__valid_match, which also applies taxon_best_match's filters, since taxon_match is now accessed through taxon_best_match
- 12:17 AM Revision 13878: fix: inputs/.TNRS/schema.sql: MatchedTaxon: use taxon_best_match instead of taxon_match because this should provide only one match per taxon
07/02/2014
- 02:17 AM Revision 13877: bugfix: /README.TXT: Mac settings backup: backup to jupiter: need to exclude ~/software/**/.svn/ because these are different on jupiter
- 02:00 AM Revision 13876: /README.TXT: Mac settings backup: backup to jupiter: removed no longer applicable exclude of /VirtualBox VMs/Ubuntu/Ubuntu.vdi
06/27/2014
- 03:15 PM Task #917: TNRS: Instructions for new version with TPL
- test update now that watcher notifications have been fixed
06/26/2014
- 06:30 PM Revision 13875: inputs/.TNRS/schema.sql: added taxon_best_match view
- 06:12 PM Revision 13874: inputs/.TNRS/schema.sql: taxon_match: added taxon_match__one_selected_match unique index
- 05:06 PM Revision 13873: backups/TNRS.backup.md5: updated
- 09:49 AM Revision 13872: planning/meetings/BIEN conference call availability.xlsx: updated
- 09:25 AM Task #917: TNRS: Instructions for new version with TPL
- testing Martha's watcher notifications again after e-mail address fix...
- 09:23 AM Task #917: TNRS: Instructions for new version with TPL
- testing Martha's watcher notifications...
- 05:41 AM Revision 13871: inputs/.TNRS/schema.sql: taxon_match__fill(): split into separate DECLARE blocks for each field for clarity
- 04:58 AM Revision 13870: inputs/.TNRS/data.sql: refreshed
- 04:57 AM Revision 13869: fix: inputs/.TNRS/schema.sql: taxon_match: renamed related items to start with taxon_match__*
- 04:33 AM Revision 13868: fix: inputs/.TNRS/schema.sql: taxon_match: renamed related items to start with taxon_match__*
- 04:28 AM Revision 13867: inputs/.TNRS/data.sql: refreshed
- 04:11 AM Revision 13866: inputs/.TNRS/schema.sql: taxon_match: insert names via taxon_match_input auto-updatable view instead of directly into taxon_match, to allow the taxon_match columns to be renamed while still supporting inserts using the TNRS column names
- 02:43 AM Revision 13865: fix: schemas/Makefile: vegbien.sql: also need to update inputs/.TNRS/data.sql, since its contents change along with this
- 02:41 AM Revision 13864: inputs/.TNRS/schema.sql: tnrs_match: renamed to taxon_match to use the normalized VegCore name for this, and to avoid repeating the schema name
- 02:39 AM Revision 13863: inputs/.TNRS/schema.sql: tnrs_match: renamed to taxon_match to use the normalized VegCore name for this, and to avoid repeating the schema name
- 02:38 AM Revision 13862: inputs/.TNRS/schema.sql: tnrs_match: renamed to taxon_match to use the normalized VegCore name for this, and to avoid repeating the schema name
- 02:14 AM Revision 13861: inputs/.TNRS/schema.sql: tnrs_match: renamed to taxon_match to use the normalized VegCore name for this, and to avoid repeating the schema name
Also available in: Atom