Activity
From 06/28/2014 to 07/27/2014
07/27/2014
- 03:19 PM Revision 14373: bugfix: schemas/util.sql: derived_cols_export(): also need to first run util.derived_cols_remove() if changing the derived column order
- 03:15 PM Revision 14372: schemas/util.sql: added derived_cols_remove(), which allows derived columns to be re-created in a different order
- 03:01 PM Revision 14371: schemas/util.sql: added drop_derived_col(), which ensures that drop_column() only cascades to views
- 02:54 PM Revision 14370: fix: schemas/util.sql: drop_column(): also need to recreate dependent views
- 02:26 PM Revision 14369: schemas/util.sql: added drop_constraint()
07/26/2014
- 10:16 PM Revision 14368: inputs/.TNRS/schema.sql: taxon_match: `inputs/.TNRS/data.sql.run refresh`: documented runtime (1 min)
- 10:15 PM Revision 14367: bugfix: inputs/.TNRS/schema.sql: taxon_match: use "Accepted_species[_binomial]__@TNRS__@vegpath.org" instead of "*Accepted_name_species". this fixes a bug in __accepted_infraspecific_label where Accepted_name_species with trailing whitespace could not be prefix-removed from names that contained just a species binomial.
- 09:57 PM Revision 14366: fix: inputs/.TNRS/schema.sql: taxon_match: added derived column "Accepted_species[_binomial]__@TNRS__@vegpath.org", which removes trailing whitespace
- 08:58 PM Revision 14365: inputs/.TNRS/schema.sql: added steps to remove a column and to add a non-derived column
- 08:48 PM Revision 14364: inputs/.TNRS/schema.sql: taxon_match: to remove columns or add columns at the end: merged into "to add a new derived column"
- 08:44 PM Revision 14363: inputs/.TNRS/schema.sql: to add columns in the middle: renamed to "to move a column to the middle" for clarity
- 08:38 PM Revision 14362: inputs/.TNRS/schema.sql: to populate a new column: updated to use util.derived_col_update()
- 08:24 PM Revision 14361: fix: inputs/.TNRS/schema.sql: taxon_match: to remove columns or add columns: also need to run util.recreate_view()
- 08:19 PM Revision 14360: inputs/.TNRS/schema.sql: taxon_match: to remove columns or add columns at the end: don't need to run `rm=1 inputs/.TNRS/data.sql.run` because this is now run by `make schemas/remake`
- 08:13 PM Revision 14359: inputs/.TNRS/schema.sql: added new derived columns to derived views
- 07:52 PM Revision 14358: schemas/util.sql: remove_prefix(), remove_suffix(): support case-insensitive matching
- 07:43 PM Revision 14357: bugfix: inputs/.TNRS/schema.sql: taxon_match.__accepted_infraspecific_label: need to use case-insensitive matching of the removed prefix because TNRS lowercases part of the Accepted_name
- 07:40 PM Revision 14356: schemas/util.sql: remove_prefix(), remove_suffix(): support case-insensitive matching
- 06:54 PM Revision 14355: schemas/util.sql: remove_prefix(), remove_suffix(): support case-insensitive matching
- 06:52 PM Revision 14354: schemas/util.sql: added case-sensitive-optional versions of starts_with(), ends_with()
- 06:50 PM Revision 14353: schemas/util.sql: added canon_case(str text, case_sensitive boolean)
- 06:36 PM Revision 14352: schemas/util.sql: check_constraint_disable(): documented that we can't use ADD CONSTRAINT NOT VALID for this because it does not disable the constraint completely, but instead causes new rows to be validated against it. the ADD CONSTRAINT NOT VALID behavior does not work for the derived columns use case of this function.
- 06:17 PM Revision 14351: schemas/public_.sql: *_view_modify(): added repopulate param, which is passed to rematerialize_view()
- 06:14 PM Revision 14350: schemas/util.sql: rematerialize_view(): support updating just the table's schema, without populating it
- 05:37 PM Revision 14349: schemas/util.sql: rematerialize_view(): don't need extra eval() around table_mod_sql because recreate() performs eval()
- 05:28 PM Revision 14348: lib/sh/sync.sh: db_snapshot(): before backing up, trim bloated temp files (eg. from rolled back changes)
- 05:16 PM Revision 14347: bugfix: inputs/.TNRS/schema.sql: taxon_match: use wrapper for util.remove_prefix() so CHECK constraints that use it don't get dropped when the util schema is reinstalled
- 04:32 PM Revision 14346: schemas/public_.sql: *_view_modify(): use util.rematerialize_view(), which now supports table mods
- 03:47 PM Revision 14345: schemas/util.sql: rematerialize_view(): added support for table mods to run after creating table
- 04:53 AM Revision 14344: bugfix: schemas/util.sql: rematerialize_view(): don't populate unless recreate() can successfully recreate the dependent views, to avoid populating the entire table (which for some views can take hours) only to have the changes rolled back by a failing dependent view
- 04:42 AM Revision 14343: schemas/util.sql: removed no longer used materialize_view(). use util.copy() instead.
- 04:41 AM Revision 14342: schemas/util.sql: rematerialize_view(): call util.copy() directly instead of via util.materialize_view()
- 04:35 AM Revision 14341: schemas/util.sql: removed no longer used copy_types_and_data(). use util.materialize_view() instead.
- 04:20 AM Revision 14340: schemas/util.sql: to_freq(): use util.copy() instead of util.copy_types_and_data() to avoid a dependency on util.copy_types_and_data()
07/25/2014
- 09:57 PM Revision 14339: fix: schemas/util.sql: copy_data(): also need to ANALYZE table after loading data into it
- 09:13 PM Revision 14338: schemas/public_.sql: analytical_stem_view: use viewFullOccurrence_individual_view to avoid duplicated code between the views. this moves the stem-related columns to the end of the view.
- 06:26 PM Revision 14337: schemas/public_.sql: added viewFullOccurrence_individual_view
- 06:20 PM Revision 14336: web/username_prefix.php: substitutions: also support "" because Postgres doubles " to escape them
- 06:17 PM Revision 14335: web/username_prefix.php: added support for substitutions ( x."_y@url -> x.x_y@url ), which can be used when a table name is repeated in a column name
- 06:15 PM Revision 14334: backups/TNRS.backup.md5: updated
- 02:31 PM Revision 14333: schemas/util.sql: derived_col_constraint_initially_enabled(): clarified that this is always based on whether the existing constraint is enabled
- 02:21 PM Revision 14332: inputs/.TNRS/schema.sql: taxon_match: COMMENT: added steps to port derived column changes to vegbiendev
- 02:12 PM Revision 14331: schemas/util.sql: added derived_cols_export(), which creates SQL to port the derived cols to another DB
- 12:41 PM Revision 14330: bugfix: inputs/.TNRS/schema.sql: taxon_match: derived columns: use new "matched~Name[_no_author]___@TNRS__@vegpath.org" instead of "*Name_matched" so that "No suitable matches found." is removed before concatenating with other fields
- 12:14 PM Revision 14329: inputs/.TNRS/schema.sql: taxon_match: added derived column "matched~Name[_no_author]___@TNRS__@vegpath.org", which removes the "No suitable matches found." string
07/22/2014
- 06:50 AM Revision 14328: inputs/.TNRS/schema.sql: reordered derived columns in dependency order
- 06:46 AM Revision 14327: bugfix: inputs/.TNRS/schema.sql: "[accepted_]morphospecies[_binomial]__@Brad__.TNRS@vegpath.org": need to use "[accepted_]genus__@DwC__@vegpath.org" rather than "*Accepted_name" for this for rank = genus
- 06:44 AM Revision 14326: inputs/.TNRS/schema.sql: taxon_match: added derived column "[scrubbed_]morphospecies[_binomial]__@Brad__.TNRS@vegpath.org"
- 06:41 AM Revision 14325: bugfix: inputs/.TNRS/schema.sql: "[accepted_]genus__@DwC__@vegpath.org": need to populate this for rank = genus
- 05:47 AM Revision 14324: inputs/.TNRS/schema.sql: taxon_match: added derived column "[scrubbed_]taxonomicStatus__@DwC__@vegpath.org"
- 05:30 AM Revision 14323: bugfix: inputs/.TNRS/schema.sql: derived columns: use "Accepted_family__@TNRS__@vegpath.org" instead of "*Accepted_name_family" because "*Accepted_name_family" is sometimes missing
- 05:24 AM Revision 14322: bugfix: derived_col_constraints_enable(): also need to update col comments, in addition to table__fill_derived()
- 04:54 AM Revision 14321: schemas/util.sql: check_constraint_enabled(): made it IMMUTABLE
- 04:48 AM Revision 14320: bugfix: schemas/util.sql: derived_col_constraints_enable(): need to update table__fill_derived() w/ formatting changes from enabling constraints
- 04:40 AM Revision 14319: bugfix: schemas/util.sql: derived_col_constraints_enable(): need to update table__fill_derived() w/ formatting changes from enabling constraints
- 04:12 AM Revision 14318: fix: inputs/.TNRS/schema.sql: taxon_match: added derived column "Accepted_family__@TNRS__@vegpath.org", which is needed because "*Accepted_name_family" isn't always populated
- 04:01 AM Revision 14317: bugfix: inputs/.TNRS/schema.sql: taxon_match: to add columns in the middle: also need to run util.derived_cols_repopulate() since the dependency order has changed
- 03:57 AM Revision 14316: fix: schemas/util.sql: derived_cols_populate(): documented that this function will sometimes not repopulate columns when needed, such as after changing the column order. in these cases, it is necessary to run util.derived_cols_repopulate() manually.
- 03:53 AM Revision 14315: schemas/util.sql: derived_cols_populate(): use new derived_cols_repopulate()
- 03:51 AM Revision 14314: schemas/util.sql: added derived_cols_repopulate()
- 03:25 AM Revision 14313: fix: inputs/.TNRS/schema.sql: taxon_match: COMMENT: to add columns in the middle: also need to run util.derived_cols_update()
- 03:23 AM Revision 14312: fix: inputs/.TNRS/schema.sql: taxon_match: COMMENT: updated util.derived_cols_sync() to util.derived_cols_update()
- 03:20 AM Revision 14311: inputs/.TNRS/schema.sql: reordered derived columns in dependency order
- 03:12 AM Revision 14310: inputs/.TNRS/schema.sql: reordered derived columns in dependency order
- 02:44 AM Revision 14309: bugfix: schemas/util.sql: derived_cols_trigger_update(): need to clear derived cols first so old values won't be used in calculations
- 02:36 AM Revision 14308: inputs/.TNRS/schema.sql: derived_cols_trigger_update(): put expr on same line as var, to save space
- 02:33 AM Revision 14307: inputs/.TNRS/schema.sql: derived_cols_trigger_update(): removed lines between statements, which add too much space
- 02:27 AM Revision 14306: fix: schemas/util.sql: derived_cols_trigger_update(): don't indent expr because may have multiple lines
- 02:25 AM Revision 14305: bugfix: schemas/util.sql: derived_col_update(): set_comment(): vars can't have same name as params, which will be substituted
- 01:48 AM Revision 14304: inputs/.TNRS/schema.sql: taxon_match: added derived column "[accepted_]morphospecies[_binomial]__@Brad__.TNRS@vegpath.org"
- 01:37 AM Revision 14303: inputs/.TNRS/schema.sql: taxon_match: added derived column "[parsed_]morphospecies[_suffix]__@Brad__.morphosp@vegpath.org"
- 01:27 AM Revision 14302: added web/people/Brad_Boyle/morphosp symlink to morphospecies
- 12:37 AM Revision 14301: added derived/TNRS/Morphospecies and scope.eml* from Brad
- 12:36 AM Revision 14300: added web/people/Brad_Boyle/morphospecies redirect to Morphospecies and scope.eml.txt
- 12:35 AM Revision 14299: added web/people/Brad_Boyle/Morphospecies and scope.eml.txt symlink
- 12:34 AM Revision 14298: added derived/TNRS/Morphospecies and scope.eml* from Brad
- 12:27 AM Revision 14297: added web/people/Brad_Boyle/TNRS redirect to Modifications to procedure to scrubbing names using TNRS.docx
- 12:27 AM Revision 14296: added web/people/Brad_Boyle/Modifications to procedure to scrubbing names using TNRS.docx symlink
- 12:18 AM Revision 14295: added derived/TNRS/Modifications to procedure to scrubbing names using TNRS.docx from Brad
07/21/2014
- 11:59 PM Revision 14294: schemas/public_.sql: analytical_stem: synced to analytical_stem_view
- 11:56 PM Revision 14293: bugfix: schemas/util.sql: view_body(): need to match view defs without trailing \n
- 11:53 PM Revision 14292: bugfix: schemas/util.sql: view_body(): need to support both CREATE statements and view bodies
- 11:23 PM Revision 14291: bugfix: schemas/util.sql: raise(): use eval_raw() so that functions called by eval() (such as view_def_to_orig()) can use raise() for debugging
- 09:30 PM Revision 14290: schemas/util.sql: eval(): use new eval_raw()
- 09:29 PM Revision 14289: schemas/util.sql: added eval_raw()
- 07:52 PM Revision 14288: bugfix: schemas/util.sql: view_is_automatically_updatable(), view_is_subset(): use util.view_body() so not matching COMMENT statements, etc
- 07:35 PM Revision 14287: schemas/util.sql: added view_body()
- 07:03 PM Revision 14286: bugfix: schemas/util.sql: view_is_subset(): need to support "(-x-)" in col name
- 06:34 PM Revision 14285: schemas/util.sql: added copy_struct(from_ regtype, to_ text) and copy_data(from_ regtype, to_ text), and use these in a general-purpose copy()
- 05:44 PM Revision 14284: inputs/.TNRS/schema.sql: added new derived columns to derived views
- 05:30 PM Revision 14283: fix: schemas/util.sql: derived_col_update(): column comment: formula: put on own line to support multiline exprs
- 05:26 PM Revision 14282: bugfix: schemas/util.sql: derived_col_update(): column comment: need to use expr from DB, not as provided by user, to avoid this changing when derived_col_update() is rerun
- 05:13 PM Revision 14281: inputs/.TNRS/schema.sql: taxon_match: added derived columns scrubbed_*
- 05:03 PM Revision 14280: bugfix: schemas/util.sql: derived_col_expr_from_check_constraint(): need to handle strs with \n after IS DISTINCT FROM
- 04:34 PM Revision 14279: inputs/.TNRS/schema.sql: taxon_match: put matched-taxon derived columns before accepted-taxon derived columns
- 04:25 PM Revision 14278: inputs/.TNRS/schema.sql: taxon_match: added derived column "[matched_]species[_binomial]~(-Accepted_-)__@TNRS__@vegpath.org"
- 04:08 PM Revision 14277: inputs/.TNRS/schema.sql: _matched_has_accepted: renamed to matched_has_accepted because the leading _ did not create vertical alignment anyway, due to different quoting syntaxes
- 04:05 PM Revision 14276: schemas/util.sql: added derived_col_name_from_check_constraint()
- 03:48 PM Revision 14275: schemas/util.sql: derived_col_update(): add steps to rename column
- 03:42 PM Revision 14274: web/username_prefix.php: transposes: removed recommendation for vertical alignment because different quoting syntaxes, etc. make it difficult to have a consistent alignment anyway
- 03:34 PM Revision 14273: bugfix: schemas/util.sql: derived_col_update(): don't set_comment() until CHECK constraint exists so that we can use its canon-ed formula
- 03:24 PM Revision 14272: bugfix: schemas/util.sql: check_constraint_expr_enable(): don't remove suffix without also removing prefix. this is done by using extract_by_regexp() instead. also changed check_constraint_enabled() to use regexps to match this.
- 03:08 PM Revision 14271: bugfix: schemas/util.sql: extract_by_regexp(): needs surrounding COALESCE() in case no match
- 03:05 PM Revision 14270: schemas/util.sql: added extract_by_regexp()
- 09:02 AM Revision 14269: inputs/.TNRS/schema.sql: taxon_match: added derived column _matched_has_accepted
- 08:47 AM Revision 14268: inputs/.TNRS/schema.sql: added new derived columns to derived views
- 08:44 AM Revision 14267: inputs/.TNRS/schema.sql: taxon_match: added derived columns "[matched_]scientificName[_with_author]__@DwC__@vegpath.org", "[accepted_]scientificName[_with_author]__@DwC__@vegpath.org"
- 08:29 AM Revision 14266: inputs/.TNRS/schema.sql: taxon_match: added derived columns "[accepted_]Infraspecific_rank[_abbr]__@TNRS__@vegpath.org", "[accepted_]infraspecificEpithet__@DwC__@vegpath.org"
- 08:26 AM Revision 14265: inputs/.TNRS/schema.sql: taxon_match: added derived column "__accepted_infraspecific_{rank,epithet}"
- 08:19 AM Revision 14264: inputs/.TNRS/schema.sql: taxon_match: added derived column __accepted_infraspecific_label
- 08:12 AM Revision 14263: bugfix: schemas/util.sql: canon_sql(): need to use the right search_path
- 08:12 AM Revision 14262: bugfix: schemas/util.sql: check_constraint view: def: must use our wrapper so that the right search_path is used
- 08:08 AM Revision 14261: schemas/util.sql: added pg_get_expr() wrapper, which sets search_path to pg_temp
- 07:41 AM Revision 14260: bugfix: schemas/util.sql: check_constraint_expr_*(): updated strings for new Mac Postgres update, which was installed on reboot
- 07:40 AM Revision 14259: bugfix: schemas/util.sql: check_constraint_expr_*(): updated strings for new Mac Postgres update, which was installed on reboot
- 06:55 AM Revision 14258: bugfix: schemas/util.sql: derived_col_update(): steps to modify: also need to run util.derived_cols_populate()
- 06:54 AM Revision 14257: inputs/.TNRS/schema.sql: taxon_match: added derived columns "[accepted_]genus__@DwC__@vegpath.org", "[accepted_]specificEpithet__@DwC__@vegpath.org"
- 06:50 AM Revision 14256: bugfix: schemas/util.sql: check_constraint view: can't use consrc because this doesn't get auto-updated with column renames
- 06:37 AM Revision 14255: bugfix: schemas/util.sql: derived_col_constraints_enable(): need final `SELECT NULL::void` so don't fold away functions called in previous query
- 06:26 AM Revision 14254: inputs/.TNRS/schema.sql: taxon_match: ran derived_cols_update()
- 06:19 AM Revision 14253: bugfix: derived_col_constraint_initially_enabled(): must be both already enabled *and* unchanged to be initially enabled (r14240 was only partially correct)
- 06:08 AM Revision 14252: schemas/util.sql: derived_cols_trigger_update(): to regenerate instructions: use util.derived_cols_update(), which also performs other updates
- 06:07 AM Revision 14251: schemas/util.sql: derived_cols_sync(): renamed to derived_cols_trigger_update() for clarity
- 06:05 AM Revision 14250: schemas/util.sql: added derived_cols_update()
- 05:58 AM Revision 14249: schemas/util.sql: derived_cols_sync(): renamed to derived_cols_trigger_update() for clarity
- 05:54 AM Revision 14248: schemas/util.sql: derived_col_update(): add comment describing how to modify derived column
- 05:41 AM Revision 14247: schemas/util.sql: added quote_typed(derived_col_def)
- 05:37 AM Revision 14246: schemas/util.sql: added quote_typed(col)
- 05:14 AM Revision 14245: schemas/util.sql: derived_cols_sync(): table__fill_derived(): document in the function comment how to regenerate it
- 05:08 AM Revision 14244: schemas/util.sql: added mk_set_comment(regprocedure, text)
- 05:07 AM Revision 14243: schemas/util.sql: set_comment(): accept anything with a mk_set_comment()
- 03:39 AM Revision 14242: fix: schemas/util.sql: check_constraint_expr_disable(): need to use CASE instead of `true OR ` because OR always evaluates both sides, slowing down constraint evaluation
- 03:23 AM Revision 14241: fix: schemas/util.sql: check_constraint view: conrelid: don't cast to regclass so can use index scan
- 03:21 AM Revision 14240: bugfix: derived_col_constraint_initially_enabled(): use whether existing constraint is already enabled, not whether expr is unchanged (which does not affect the *enabled* status)
- 02:52 AM Revision 14239: schemas/util.sql: check_constraint_canon_sql(): use util.debug_print_return_value() to assist debugging
- 02:49 AM Revision 14238: bugfix: schemas/util.sql: check_constraint_replace(): need to use util.check_constraint_canon_sql() instead of util.canon_sql()
- 02:40 AM Revision 14237: schemas/util.sql: added check_constraint_canon_sql(). this must be used instead of plain util.canon_sql() because unlike queries, CHECK constraints are not simplified, just standardized.
- 02:06 AM Revision 14236: schemas/util.sql: check_constraint_replace(): support not canon-ing def.expr
- 12:23 AM Revision 14235: bugfix: schemas/util.sql: CHECK constraint exprs: need to remove the enclosing () to be consistent everywhere
07/20/2014
- 11:59 PM Revision 14234: bugfix: schemas/util.sql: canon_sql(): need to include the enclosing () because the DB includes them
- 11:42 PM Revision 14233: fix: schemas/util.sql: canon_sql(): use explain()'s verbose mode to facilitate debugging
- 11:38 PM Revision 14232: bugfix: schemas/util.sql: explain(): need to restore NOTICEs in verbose mode
- 11:22 PM Revision 14231: schemas/util.sql: explain(): support running in verbose mode
- 11:19 PM Revision 14230: bugfix: schemas/util.sql: check_constraint_replace(): need to canon the expr, using new util.canon_sql(), so it can be compared with the expr in the DB
- 11:18 PM Revision 14229: schemas/util.sql: added canon_sql(expr text, table_ regclass)
- 10:45 PM Revision 14228: fix: schemas/util.sql: check_constraint view: *don't* use pretty-print mode, because this format does not match util.canon_sql()
- 10:43 PM Revision 14227: fix: schemas/util.sql: check_constraint_expr_enable(), derived_col_expr_from_check_constraint(): trim parens in order to support non-pretty-print mode, which is needed with canon_sql()
- 09:27 PM Revision 14226: schemas/util.sql: added canon_sql(expr text)
- 09:21 PM Revision 14225: schemas/util.sql: added trim_parens()
- 08:19 PM Revision 14224: schemas/util.sql: added copy_struct(from_ regtype, to_ text) and copy_data(from_ regtype, to_ text), and use these in a general-purpose copy()
- 08:02 PM Revision 14223: bugfix: schemas/util.sql: copy(from_ regtype, to_ text): must be declared STRICT to handle NULL properly
- 07:58 PM Revision 14222: schemas/util.sql: added copy(from_ regtype, to_ text)
- 07:39 PM Revision 14221: fix: schemas/util.sql: is_raise_condition(): should be called is_log_level()
- 07:36 PM Revision 14220: schemas/util.sql: raise(): support SQLSTATE values
- 07:33 PM Revision 14219: bugfix: schemas/util.sql: raise_expr(raise_type): only treat as raise condition if not SQLSTATE, to allow custom SQLSTATEs
- 07:30 PM Revision 14218: schemas/util.sql: raise(): support SQLSTATE values
- 07:28 PM Revision 14217: schemas/util.sql: added raise_expr(raise_type)
- 07:28 PM Revision 14216: schemas/util.sql: added is_raise_condition()
- 07:14 PM Revision 14215: schemas/util.sql: added is_sqlstate()
- 06:44 PM Revision 14214: schemas/util.sql: typeof(expr text, table_ regtype): use simpler util.eval2val() instead of EXECUTE
- 06:13 PM Revision 14213: fix: schemas/util.sql: derived_col_constraint_initially_enabled(): also should be initially enabled if expr unchanged, to avoid re-running constraint on all rows
- 06:09 PM Revision 14212: schemas/util.sql: derived_col_update(): use new util.derived_col_constraint_initially_enabled()
- 06:09 PM Revision 14211: schemas/util.sql: added derived_col_constraint_initially_enabled()
- 06:05 PM Revision 14210: schemas/util.sql: added derived_col_expr(col)
- 05:56 PM Revision 14209: derived_col_constraint_enable(): documented that this now avoids re-running constraint on all rows if already enabled
- 05:43 PM Revision 14208: fix: check_constraint_replace(): avoid re-running the constraint on all rows if expr hasn't changed
- 05:40 PM Revision 14207: schemas/util.sql: check_constraint_replace(): shortened param to `def` for readability
- 05:39 PM Revision 14206: schemas/util.sql: added check_constraint_expr(constraint_ table_item)
- 04:34 PM Revision 14205: schemas/util.sql: added derived_col_update()
- 04:31 PM Revision 14204: schemas/util.sql: check_constraint__disabled(): made disabled configurable (renaming func to check_constraint())
- 04:20 PM Revision 14203: schemas/util.sql: added derived_col_constraint_expr()
- 04:07 PM Revision 14202: schemas/util.sql: derived_col_constraints_enable(): use new derived_col_constraint_enable()
- 04:06 PM Revision 14201: schemas/util.sql: added derived_col_constraint_enable()
- 03:52 PM Revision 14200: schemas/util.sql: added table_has_data()
- 03:47 PM Revision 14199: schemas/util.sql: added col_add(derived_col_def)
- 03:44 PM Revision 14198: schemas/util.sql: added typeof(text, regclass)
- 03:40 PM Revision 14197: bugfix: /README.TXT: Full database import: added warnings that you should not run backups/pg_snapshot while the import is running, nor until the previous import has been replaced
- 03:22 PM Revision 14196: schemas/util.sql: added regtype(regclass)
- 02:52 PM Revision 14195: schemas/util.sql: added col_add()
- 02:32 PM Revision 14194: schemas/util.sql: added col_def type
- 07:22 AM Revision 14193: schemas/util.sql: derived_cols_populate(): only run if needed
- 07:09 AM Revision 14192: schemas/util.sql: added derived_cols_populated()
- 07:09 AM Revision 14191: schemas/util.sql: added derived_cols_populated()
- 07:08 AM Revision 14190: schemas/util.sql: added derived_col_populated()
- 07:07 AM Revision 14189: schemas/util.sql: added check_constraint_enabled()
- 06:46 AM Revision 14188: schemas/util.sql: remove_prefix(), remove_suffix(): use new starts_with(), ends_with()
- 06:45 AM Revision 14187: schemas/util.sql: added starts_with(), ends_with()
- 05:57 AM Revision 14186: schemas/util.sql: derived_cols_populate(): also run util.derived_col_constraints_enable() once populated
- 05:55 AM Revision 14185: schemas/util.sql: added derived_col_constraints_enable()
- 05:28 AM Revision 14184: schemas/util.sql: added check_constraint__disabled(), which creates a CHECK constraint which is initially disabled
- 05:27 AM Revision 14183: schemas/util.sql: added check_constraint__disabled(), which creates a CHECK constraint which is initially disabled
- 05:07 AM Revision 14182: fix: schemas/util.sql: run_triggers(): need to re-sort rows if table needs this
- 05:05 AM Revision 14181: schemas/util.sql: added try_cluster()
- 04:05 AM Revision 14180: schemas/util.sql: added derived_cols_populate()
- 04:04 AM Revision 14179: schemas/util.sql: added run_triggers(regclass)
- 04:00 AM Revision 14178: schemas/util.sql: added first_col(regclass)
- 02:42 AM Revision 14177: fix: schemas/util.sql: read-only functions should be STABLE rather than VOLATILE
- 02:42 AM Revision 14176: bugfix: schemas/util.sql: functions that access the DB must be STABLE rather than IMMUTABLE
- 02:23 AM Revision 14175: bugfix: schemas/util.sql: derived_col_defs(): derived columns *must* be returned in table order (which should = dependency order), so that they are populated in dependency order
- 02:17 AM Revision 14174: bugfix: schemas/util.sql: remove_prefix(), remove_suffix(): must be declared STRICT to handle NULL properly
- 02:16 AM Revision 14173: bugfix: schemas/util.sql: remove_suffix(): need to use quote_nullable(str) so a NULL str doesn't NULL out the entire message
- 02:10 AM Revision 14172: bugfix: schemas/util.sql: remove_prefix(): need to use quote_nullable(str) so a NULL str doesn't NULL out the entire message
- 02:09 AM Revision 14171: bugfix: schemas/util.sql: raise(): use new quote_as_str() to avoid "RAISE statement option cannot be null" errors
- 02:08 AM Revision 14170: schemas/util.sql: added quote_as_str()
07/19/2014
- 11:23 PM Revision 14169: schemas/util.sql: added CAST (util.col AS util.derived_col_def)
- 11:05 PM Revision 14168: schemas/util.sql: derived_col_def(col): use existing util.check_constraint_def()
- 11:01 PM Revision 14167: schemas/util.sql: added CAST (util.col AS util.table_item)
- 10:50 PM Revision 14166: schemas/util.sql: added derived_col_def(col)
- 10:30 PM Revision 14165: schemas/util.sql: added check_constraint_disable()
- 10:23 PM Revision 14164: bugfix: schemas/util.sql: check_constraint_drop(): need IF EXISTS
- 10:09 PM Revision 14163: schemas/util.sql: added check_constraint_expr_disable()
- 09:50 PM Revision 14162: schemas/util.sql: check_constraint_enable(): documented that this is idempotent
- 09:21 PM Revision 14161: inputs/.TNRS/schema.sql: taxon_match: documented that whenever columns are renamed, util.derived_cols_sync() should be run
- 09:14 PM Revision 14160: schemas/util.sql: derived_cols_sync(): also create trigger, since this is necessary for a complete sync
- 09:13 PM Revision 14159: schemas/util.sql: added name(regprocedure)
- 09:02 PM Revision 14158: schemas/util.sql: derived_cols_sync(): set the trigger name in a var
- 08:52 PM Revision 14157: schemas/util.sql: added check_constraint_enable()
- 08:46 PM Revision 14156: schemas/util.sql: added check_constraint_replace()
- 08:43 PM Revision 14155: schemas/util.sql: added CAST (util.table_item AS util.check_constraint_def)
- 08:38 PM Revision 14154: schemas/util.sql: added check_constraint_drop()
- 08:02 PM Revision 14153: schemas/util.sql: derived_col_expr_from_check_constraint(): enable initially-disabled exprs
- 08:00 PM Revision 14152: schemas/util.sql: added check_constraint_expr_enable()
- 07:42 PM Revision 14151: fix: schemas/util.sql: check_constraint view: def: use pg_get_expr() with pretty-print, to avoid extra () that complicate parsing
- 06:54 PM Revision 14150: inputs/.TNRS/schema.sql: taxon_match__fill_derived(): updated using new util.derived_cols_sync()
- 06:50 PM Revision 14149: schemas/util.sql: added derived_cols_sync()
- 06:45 PM Revision 14148: bugfix: CAST (util.check_constraint_def AS util.derived_col_def): need to extract the derived column expr *from* the CHECK constraint expr
- 06:44 PM Revision 14147: schemas/util.sql: added derived_col_expr_from_check_constraint()
- 06:39 PM Revision 14146: schemas/util.sql: added extract_str()
- 06:34 PM Revision 14145: schemas/util.sql: added remove_suffix()
- 06:21 PM Revision 14144: schemas/util.sql: added remove_prefix()
- 05:47 PM Revision 14143: schemas/util.sql: added CAST (util.table_item AS util.col)
- 05:37 PM Revision 14142: schemas/util.sql: derived_col_defs(): return derived_col_def instead of check_constraint_def
- 05:36 PM Revision 14141: schemas/util.sql: added derived_col_def type with cast from util.check_constraint_def
- 04:51 PM Revision 14140: schemas/util.sql: added derived_col_defs()
- 04:35 PM Revision 14139: schemas/util.sql: check_constraint_defs(): use new util.check_constraint view
- 04:30 PM Revision 14138: schemas/util.sql: added check_constraint view
- 03:53 PM Revision 14137: schemas/util.sql: added check_constraint_defs()
- 03:53 PM Revision 14136: schemas/util.sql: added check_constraint_def type
- 03:51 PM Revision 14135: schemas/util.sql: added table_item type
- 01:59 PM Revision 14134: schemas/util.sql: col_ref: renamed to just col
- 01:38 PM Revision 14133: web/username_prefix.php: remove padding used to visually separate elements: documented that this leaves embedded __ (eg. x__y) untouched
- 02:52 AM Revision 14132: fix: inputs/.TNRS/schema.sql: taxon_match: to populate a new column: also need to recluster table so rows are exported in sorted order
- 02:47 AM Revision 14131: schemas/util.sql: cluster(): support omitting index to recluster a table
- 02:26 AM Revision 14130: schemas/util.sql: cluster_once(): use util.cluster()
- 02:24 AM Revision 14129: schemas/util.sql: cluster_once(): documented that this runs cluster only *once* if called repeatedly
- 02:24 AM Revision 14128: schemas/util.sql: added cluster(), which reclusters each time
- 02:10 AM Revision 14127: schemas/util.sql: cluster_once(): use simpler `LANGUAGE sql` now that we're using util.eval() instead of EXECUTE
- 02:03 AM Revision 14126: schemas/util.sql: cluster_once(): use util.eval() instead of EXECUTE to print the CLUSTER statement (if any) it's using
07/18/2014
- 11:40 PM Revision 14125: fix: inputs/.TNRS/schema.sql: taxon_match: cluster the table so the rows are always exported in the same order, even when an UPDATE statement is used to run triggers
- 11:28 PM Revision 14124: bugfix: schemas/util.sql: cluster_once(): need to use just the name of the index, without the schema prefix
- 05:00 PM Revision 14123: schemas/util.sql: cluster_once(): use simpler IS DISTINCT FROM instead of array !=
- 03:18 PM Revision 14122: inputs/.TNRS/schema.sql: added __accepted_{genus,specific_epithet} derived column, using the steps at wiki.vegpath.org/Postgres_queries#maintaining-derived-column-formulas-across-column-renames
- 02:01 AM Revision 14121: web/username_prefix.php: remove padding used to visually separate elements: documented that this also removes a leading __ that indicates an internal field
07/17/2014
- 07:18 PM Revision 14120: web/links/index.htm: updated to Firefox bookmarks: Quick links: added VegBIEN data dictionary spreadsheet, TNRS data dictionary.
- 06:15 PM Revision 14119: fix: web/username_prefix.php: transposes: changed recommended syntax to _y~x@url so that _y~ lines up vertically with [y_] paths
- 06:11 PM Revision 14118: web/username_prefix.php: removed special category comments syntax because this is now handled by transposes and insertion comments
- 06:10 PM Revision 14117: web/username_prefix.php: added support for transposes ( y~x_@url -> x_y@url , Y~x@url -> xY@url ), which can be used to put the category in a column name first without duplicating the category
- 05:35 PM Revision 14116: web/username_prefix.php: removed linewraps code because Google spreadsheets no longer supports wrapping on a - , and because a shorter syntax for category comments is now available
- 05:32 PM Revision 14115: web/username_prefix.php: category comments: moved _-_ outside [] in comparison example since it will be deleted separately
- 05:27 PM Revision 14114: bugfix: web/username_prefix.php: category comments: need /__/ around regexp
- 05:24 PM Revision 14113: web/username_prefix.php: category comments: added support for a shorter syntax than [x_-_]y@url : x+y@url -> y@url
- 04:14 PM Revision 14112: schemas/vegbien.ERD.mwb: updated
- 04:13 PM Revision 14111: fix: inputs/.TNRS/schema.sql: taxon_match: added back * prefixes on TNRS-namespace column names
- 03:34 PM Revision 14110: inputs/.TNRS/schema.sql: MatchedTaxon: taxon_best_match: don't alias to `s` since this is no longer a nested select
- 03:32 PM Revision 14109: inputs/.TNRS/schema.sql: MatchedTaxon: nested select: use just taxon_best_match since this no longer performs renamings
- 03:27 PM Revision 14108: inputs/.TNRS/schema.sql: MatchedTaxon: renamed output columns to match input columns, so that the nested select would not be performing any renamings
- 03:05 PM Revision 14107: fix: inputs/.TNRS/schema.sql: MatchedTaxon: added all columns from taxon_match (some of them apparently hadn't been included in MatchedTaxon)
- 03:01 PM Revision 14106: fix: inputs/.TNRS/schema.sql: reordered columns to match taxon_match
- 02:38 PM Revision 14105: fix: inputs/.TNRS/schema.sql: MatchedTaxon: added all columns from taxon_match (some of them apparently hadn't been included in MatchedTaxon)
- 01:02 PM Revision 14104: fix: inputs/.TNRS/schema.sql: added back Family_matched, which had gotten merged with Name_matched_accepted_family when TNRS temporarily stopped populating Name_matched_accepted_family
- 12:32 PM Revision 14103: inputs/.TNRS/schema.sql: MatchedTaxon: taxonomicStatus: moved outside the inner SELECT so that the inner SELECT would consist solely of renamings
- 12:12 PM Revision 14102: inputs/.TNRS/schema.sql: MatchedTaxon: removed unneeded "Name_matched." prefix on source-specific names (only the * is needed because there is only one table)
- 12:10 PM Revision 14101: bugfix: schemas/util.sql: view_is_subset(): auto-updatable views that are *not* plain subsets: include ones with WHERE clauses
- 12:03 PM Revision 14100: bugfix: schemas/util.sql: view_is_subset(): also need to handle auto-updatable views that are *not* plain subsets (there are some of these)
- 09:05 AM Revision 14099: bin/import_all: hidden_srcs(): removed `by_col=1` because these should be done in the same mode as the main datasources
07/16/2014
- 07:55 PM Revision 14098: schemas/public_.sql: removed no longer used threatened_taxonlabel. use iucn_red_list instead.
- 07:39 PM Revision 14097: fix: schemas/public_.sql: analytical_stem_view: don't use threatened_taxonlabel, which was never populated correctly. instead, this can eventually be rewritten to use new iucn_red_list.
- 07:13 PM Revision 14096: bin/make_analytical_db: removed threatened_taxonlabel because this is now handled by iucn_red_list
- 07:12 PM Revision 14095: bin/make_analytical_db: added iucn_red_list_view_modify()
- 06:48 PM Revision 14094: bin/make_analytical_db: removed unused code to create views in the analytical_db schema
- 06:35 PM Revision 14093: bin/make_analytical_db: merged mk_table and mk_analytical_table since they now do the same thing
- 06:34 PM Revision 14092: bin/make_analytical_db: removed unused code to create views in the analytical_db schema
- 05:05 PM Revision 14091: /README.TXT: Notes on system stability: added warning that when shutting down the VM, one should always first stop Postgres, to prevent the OS from SIGKILLing it
- 05:04 PM Revision 14090: inputs/SALVIAS/Source/header.csv: updated
- 03:50 PM Revision 14089: bugfix: bin/with_all: isset(): need to use `&>/dev/null` instead of `>&-`, etc because closing an fd causes declare to return false
- 03:31 PM Revision 14088: bugfix: bin/with_all, import_all: don't disown processes because they should be auto-killed if the shell is (disown was only needed before we used screen)
- 03:24 PM Revision 14087: bugfix: Full database import: need nested shell to prevent errexit from closing the window
- 01:24 PM Revision 14086: bugfix: schemas/public_.sql: analytical_stem_view_modify(): added columns: need to include COMMENT statements
- 01:21 PM Revision 14085: bugfix: schemas/util.sql: comment(element oid): need `objsubid = 0` filter so this doesn't use comments for any other objsubids
- 12:53 PM Revision 14084: schemas/public_.sql: added iucn_red_list_view_modify() and use it in iucn_red_list_view's "after updating this" instructions
- 12:47 PM Revision 14083: bugfix: schemas/util.sql: rematerialize_view(): also need to handle the target table's dependent views
- 12:38 PM Revision 14082: bugfix: schemas/util.sql: mk_drop_from_create(): need to use `IF EXISTS` so that the resulting statement also works if the target object does not yet exist
- 12:24 PM Revision 14081: schemas/util.sql: mk_drop_from_create(): also support `SELECT util.drop_*()` statements, in addition to DROP
- 11:37 AM Revision 14080: inputs/.TNRS/Source/test.xml.ref: updated
07/15/2014
- 11:14 PM Revision 14079: bugfix: /README.TXT: logs glob: don't unintentionally match . or ..
- 10:43 PM Revision 14078: inputs/VegBank/stemlocation_/test.xml.ref: updated
- 10:41 PM Revision 14077: /README.TXT: Full database import: `make test by_col=1`: updated runtime (1 h)
- 10:07 PM Revision 14076: inputs/TEX/Specimen2/test.xml.ref: updated
- 09:35 PM Revision 14075: inputs/GBIF/raw_occurrence_record_plants/test.xml.ref: updated
- 05:44 PM Revision 14074: bugfix: lib/sql_io.py: put_table(): handle_MissingCastException(): when updating join_cols, don't add *new* entry for join_cols[out_col], only update existing one. this fixes #902 (import bug), and with #902 fixed, #887 (disk space leak) should no longer occur.
- 04:49 PM Revision 14073: bin/import_all: delete_logs(): documented that `trap EXIT` doesn't run until *shell* exit
- 04:48 PM Revision 14072: bin/import_all: delete_logs(): print when this happens, so it can be verified that it's happening properly
- 04:32 PM Revision 14071: bugfix: bin/import_all: need to run delete_logs manually because `trap EXIT` doesn't run until bg cmds done
- 04:28 PM Revision 14070: bin/import_all: delete_logs: moved testing of whether to delete logs to delete_logs() so that delete_logs() can be run regardless of the $delete_logs setting
- 03:58 PM Revision 14069: bugfix: bin/import_all: delete_logs(): also need to match log filenames when n=""
- 03:11 PM Revision 14068: bugfix: /Makefile: postgres_restart-*: argument to echo always needs quotes
- 03:05 PM Revision 14067: bugfix: /Makefile: $(pg_ctl-*): tell user that if it doesn't work, need to run `rm $(macPostgresDir)/postmaster.pid` and retry
- 02:57 PM Revision 14066: /Makefile: $(pg_ctl-Darwin): documented the pg_ctl version of this command in case `brew services` doesn't work
- 02:39 PM Revision 14065: bugfix: bin/with_all: isset(): need to use `>&- 2>&-` because &> does not work with - as the dest
- 01:19 PM Revision 14064: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: determine if the revision works: use the in# of the query instead of waiting an arbitrary time, because this is more accurate in determining whether the import is successful
- 12:52 PM Revision 14063: config/VirtualBox_VMs/vegbiendev/README.TXT: regressions that occurred during the revision range: named these "concurrent regressions"
- 12:45 PM Revision 14062: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: need to update system time so wait times can be calculated correctly
- 12:22 PM Revision 14061: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: documented that the patches are specifically for regressions that occurred during the revision range, not all bugs that were fixed in that range
- 12:14 PM Revision 14060: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: waiting for query: increased time to min. 10 min because some revisions take longer
- 12:09 PM Revision 14059: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bin/import_all: added step to type y and press ENTER
- 12:07 PM Revision 14058: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to determine if the revision works, and then stop the import
- 11:27 AM Revision 14057: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12994
- 11:09 AM Revision 14056: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: use for loop to avoid needing to repeat the command
- 11:05 AM Revision 14055: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12760
- 10:54 AM Revision 14054: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: --fuzz apparently needs to be 3 even though we have only 2 lines of context
- 10:52 AM Revision 14053: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: use --fuzz to allow patching even when surrounding lines have changed
- 10:44 AM Revision 14052: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: sorted by revision #
- 10:43 AM Revision 14051: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: added r12865
- 10:33 AM Revision 14050: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: patch: need to use --force to avoid prompting user
- 10:16 AM Revision 14049: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: `svn revert`: need to use `--depth infinity` when reverting more than a file
- 10:14 AM Revision 14048: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: use `patch` instead of `svn merge` because `svn merge` sometimes inexplicably doens't apply patches
- 09:43 AM Revision 14047: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bugfixes: `svn revert` all changes instead of specific files, to support bugfixes to other files
- 09:42 AM Revision 14046: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: lib/sh/util.sh bugfix: use `svn merge` instead of `sed --in-place` because this can be used for more complex changes
- 09:41 AM Revision 14045: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: lib/sh/util.sh bugfix: documented revision
- 09:20 AM Revision 14044: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: need to apply r12872 bugfix to lib/sh/util.sh because it occurred after r12025
- 09:10 AM Revision 14043: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make` commands: use `yes|` to avoid needing to confirm prompts
- 09:08 AM Revision 14042: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make mk_db ...`: use `yes|` to avoid needing to confirm prompts
- 08:56 AM Revision 14041: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: to test a different revision: added step that if any of the commands after `svn up` are broken, try a different revision
- 08:46 AM Revision 14040: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: `make db`: need to run `schemas/reinstall` manually because older versions of the codebase just used `schemas/install`
07/14/2014
- 11:30 PM Revision 14039: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: documented the range from within the revision should be chosen
- 09:19 PM Revision 14038: fix: inputs/.IRMNG/*/: added _no_import because this datasource is only used as a lookup table and does not need to be imported/scrubbed
- 09:18 PM Revision 14037: config/VirtualBox_VMs/vegbiendev/README.TXT: `svn up`: documented runtime (1 min)
- 08:24 PM Revision 14036: bugfix: added nodes/_no_import, because the NCBI import doesn't work due to a Postgres 9.3 bug (#859). _no_import used to not be needed because we didn't import the hidden sources.
- 08:05 PM Revision 14035: fix: bin/with_all: removed debug statements
- 05:19 PM Revision 14034: bugfix: bin/with_all: testing if @inputs is set: `"${inputs+isset}"` syntax doesn't work for empty arrays, so need to use `declare -p` instead
- 04:46 PM Revision 14033: bugfix: to set up the VM: to use a database backup: also need to `make rm_initial_public` because initdb creates a non-empty DB
- 04:42 PM Revision 14032: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to test a different revision
- 04:40 PM Revision 14031: /Makefile: db: use reinstall instead of install to allow running this on an existing DB
- 02:49 PM Revision 14030: config/VirtualBox_VMs/vegbiendev/README.TXT: `sudo apt-get dist-upgrade`: documented runtime (12 min)
- 02:47 PM Revision 14029: config/VirtualBox_VMs/vegbiendev/README.TXT: updated VM snapshot names
- 02:44 PM Revision 14028: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to install system updates: added step to restart, because some system updates require this
- 02:34 PM Revision 14027: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: to install system updates: added step to restart, because some system updates require this
- 02:29 PM Revision 14026: config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions to install system updates
- 01:23 PM Revision 14025: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: bin/import_all: documented runtime (50 min)
- 01:07 PM Revision 14024: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to run the import
- 12:24 PM Revision 14023: config/VirtualBox_VMs/vegbiendev/README.TXT: to set up the VM: added steps to move vegbiendev.vdi to the internal hard drive
- 12:19 PM Revision 14022: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added steps to start the VM
- 12:16 PM Revision 14021: config/VirtualBox_VMs/vegbiendev/README.TXT: moved "set up the VM" steps to separate section
- 12:06 PM Revision 14020: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: added steps to create a booted snapshot
- 12:01 PM Revision 14019: config/VirtualBox_VMs/vegbiendev/README.TXT: moved creation of pg_restore snapshot to be part of just the database backup section
- 11:52 AM Revision 14018: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: documented runtime (15 min)
- 11:51 AM Revision 14017: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: time the operation
- 11:32 AM Revision 14016: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: added name for VM snapshot after this step
- 11:31 AM Revision 14015: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: need to run as DB superuser to restore py_util
- 11:24 AM Revision 14014: config/VirtualBox_VMs/vegbiendev/README.TXT: rsync: updated runtime (25 min)
- 11:24 AM Revision 14013: config/VirtualBox_VMs/vegbiendev/README.TXT: added steps to create snapshot before running pg_restore
- 10:55 AM Revision 14012: config/VirtualBox_VMs/vegbiendev/README.TXT: added steps to configure the VM before booting
- 10:53 AM Revision 14011: config/VirtualBox_VMs/vegbiendev/README.TXT: added instructions for selecting the initial snapshot and booting the VM
- 10:36 AM Revision 14010: config/VirtualBox_VMs/vegbiendev/README.TXT: perform long-running operations inside `screen`
- 10:35 AM Revision 14009: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to use a database backup: only run mk_db, not db, to avoid restoring schemas that are part of the backup
- 09:33 AM Revision 14008: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: schemas must be in dependency order
- 09:21 AM Revision 14007: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: don't restore analytical_db schema because we don't use it and it depends on public, which we're not restoring
- 09:16 AM Revision 14006: web/links/index.htm: updated to Firefox bookmarks: WebEx: added steps to join a test meeting. pg_restore: documented that you need to *create* the schema before you restore into it, or else you will get a cryptic error message that it's using pg_catalog instead. added Homebrew links, including instructions for setting psycopg2 up with it. added PostGIS links. username URLs: documented how to enable these in Internet Explorer. TWiki: documented what to do when moving the TWiki install dir.
- 09:06 AM Revision 14005: bugfix: lib/Firefox_bookmarks.reformat.csv: updated for new Firefox bookmarks format, which indents the <DD> tag
- 08:36 AM Revision 14004: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: pg_restore: need to *create* the schema before you restore into it, to avoid cryptic error messages
- 08:33 AM Revision 14003: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: schemas/postgresql.conf: need to decrease shared_buffers to match VM's allocated memory (1 GB)
- 08:18 AM Revision 14002: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: schemas/postgresql.conf: need to decrease shared_buffers to match VM's allocated memory (1 GB)
07/13/2014
- 11:50 PM Revision 14001: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: need to run `make db` to restore users, which are not part of the backup
- 11:40 PM Revision 14000: config/VirtualBox_VMs/vegbiendev/README.TXT: rsync: updated runtime (35 min)
07/11/2014
- 04:54 PM Revision 13999: config/VirtualBox_VMs/vegbiendev/README.TXT: turning off of non-ARIZ inputs: removed metadata datasources because these have been renamed to start with ".", and so aren't included in */
- 04:48 PM Revision 13998: bugfix: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: pg_restore: need to use for loop because pg_restore doesn't support multiple --schema options (it just uses the last one)
- 04:35 PM Revision 13997: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added step to restore the necessary schemas from the DB
- 04:32 PM Revision 13996: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: rsync: documented runtime (13 min)
- 04:11 PM Revision 13995: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added step to fetch the database backup via rsync
- 04:04 PM Revision 13994: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: moved steps to enable the user to access VirtualBox shared folders to separate section because VirtualBox shared folders currently don't work, likely because of a bug in VirtualBox Guest Additions when installed on something other than a clean Linux install
- 03:54 PM Revision 13993: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: database backup approach: added steps to enable the user to access VirtualBox shared folders
- 03:06 PM Revision 13992: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: added section for database backup approach, as alternative to clean database approach
- 10:32 AM Revision 13991: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: use ".." instead of - to indicate a range, to avoid this being confused for a minus sign
- 10:26 AM Revision 13990: bugfix: /Makefile: postgres_restart-Darwin: need to have the user run the command outside of make because it doesn't appear to do anything when run inside make (this is the same problem as on Linux)
- 10:25 AM Revision 13989: /Makefile: $(pg_ctl-Darwin): use simpler `brew services` instead of `launchctl __/homebrew.mxcl.postgresql.plist`
- 10:03 AM Revision 13988: bugfix: bin/stop_imports: also need to include `bin/after_import`
- 09:56 AM Revision 13987: inputs/.TNRS/Source/map.csv: datasetURL: updated to indicate that this uses the TNRS dev server (the actual URL is private)
- 09:46 AM Revision 13986: fix: inputs/VASCAN/Source/map.csv: observationGranularity should not be specimen because this datasource contains only taxa
- 09:13 AM Revision 13985: bugfix: bin/import_all: now that always using log files to fix output clutter, need to delete created logs if logging is turned off
- 08:45 AM Revision 13984: bugfix: bin/import_all: don't errexit if a background process is Ctrl-C'd
- 08:41 AM Revision 13983: bugfix: bin/import_all: was run without initial "." test: don't exit nonzero because this will close the subshell
- 08:38 AM Revision 13982: bugfix: bin/import_all: ensure that this is run in a subshell, which is needed so errexits don't close the terminal window
- 08:32 AM Revision 13981: bin/import_all: documented that this must be run in a subshell (obtained by running `$0`)
- 08:25 AM Revision 13980: bugfix: bin/import_all: need to always use log files for background processes
- 08:12 AM Revision 13979: fix: bin/import_all: Source/import: don't use by_col=1 for this because it's slower for small #s of rows. by_col mode is no longer needed for metadata-only tables because these tables now have a single empty row so that they also work in row-based mode.
- 08:06 AM Revision 13978: fix: bin/import_all: hidden srcs: use with_all for this to avoid needing to list every source, and to display the backgrounded command with the variables substituted
- 07:40 AM Revision 13977: bin/import_all: TNRS, geoscrub: integrated into the list of metadata sources
- 07:39 AM Revision 13976: bin/import_all: TNRS, geoscrub: use import rather than publish because the non-imported tables have now been excluded
- 07:34 AM Revision 13975: fix: inputs/.geoscrub/geoscrub_output/: added _no_import because these tables are metadata that is used in the analytical DB. this is better than relying on bin/import_all not to import these.
07/10/2014
- 07:25 PM Revision 13974: fix: bin/import_all: updated for new metadata datasource names (see issue #940)
- 07:25 PM Revision 13973: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: updated for new metadata datasource names (see issue #940)
- 06:32 PM Revision 13972: fix: inputs/IUCN/: renamed to inputs/.IUCN/ (see issue #940)
- 06:04 PM Revision 13971: fix: inputs/newWorld/: renamed to inputs/.newWorld/ (see issue #940)
- 04:59 PM Revision 13970: bugfix: inputs/IRMNG/: renamed to inputs/.IRMNG/ so that this comes before TNRS, which depends on it (this is a metadata datasource, so it can start with "."). part of issue #940.
- 03:52 PM Revision 13969: config/VirtualBox_VMs/vegbiendev/README.TXT: to test the import: set up the VM: added steps to create a VM snapshot when done
- 03:37 PM Revision 13968: config/VirtualBox_VMs/vegbiendev/README.TXT: `make`: documented runtime (15 min)
- 03:33 PM Revision 13967: fix: config/VirtualBox_VMs/vegbiendev/README.TXT: removed non-applicable step about the MySQL root password, which is not used for this procedure
- 03:32 PM Revision 13966: added config/VirtualBox_VMs/vegbiendev/README.TXT with instructions to test the import
- 12:17 PM Revision 13965: inputs/GBIF/_MySQL/.rsync_ignore: don't exclude GBIFPortalDB-*.data.sql.gz, even though this is an intermediate file, because it's better to have a backup of it locally. this was excluded in r13316 (2014-4-24) to free up disk space on the local machine.
- 10:57 AM Revision 13964: fix: inputs/Madidi/LocationObservation/postprocess.sql: parse Spanish dates
- 09:45 AM Revision 13963: validation/completeness/BIENDatesPopulated.xlsx: updated: Martha added Priority column
- 08:41 AM Revision 13962: schemas/util.sql: replace_words(): use map_words() to enable inlining the concatenated hstore, so that it doesn't need to be concatenated on *each* function call
- 08:39 AM Revision 13961: schemas/util.sql: added map_words()
- 08:16 AM Revision 13960: schemas/util.sql: added date_from_spanish()
- 08:13 AM Revision 13959: schemas/util.sql: added replace_words()
- 08:09 AM Revision 13958: schemas/util.sql: added words(text)
- 08:02 AM Revision 13957: schemas/util.sql: spanish_date_words(): switched to using hstore because this is better for the algorithm that will translate these
- 07:20 AM Revision 13956: schemas/util.sql: added spanish_date_words()
- 07:19 AM Revision 13955: schemas/util.sql: added `replacement` type
- 04:09 AM Revision 13954: bugfix: mk_set_relation_metadata(): need to include col comments as well so that these are re-created properly by recreate()
- 04:07 AM Revision 13953: schemas/util.sql: added mk_set_comments(regclass)
- 04:06 AM Revision 13952: schemas/util.sql: show_set_comment(regclass): renamed to mk_set_comment() to match other mk_set_comment() functions
- 04:01 AM Revision 13951: schemas/util.sql: added mk_set_col_comments(regclass)
- 03:58 AM Revision 13950: schemas/util.sql: added cols(regclass)
- 03:56 AM Revision 13949: schemas/util.sql: added mk_set_comment(col_ref)
- 03:18 AM Revision 13948: added validation/completeness/BIENDatesPopulated.xlsx from Google spreadsheets
07/09/2014
- 02:50 PM Revision 13947: fix: mappings/VegCore-VegBIEN.csv: mapped DwC year/month/day, which are split-date alternatives to eventDate. this fixes the missing eventDate in FIA.
- 02:45 PM Revision 13946: bugfix: inputs/FIA/*/VegBIEN.csv: regenerated, replacing these with symlinks to the file used by new-style import
- 02:34 PM Revision 13945: inputs/FIA/taxon_observation.**/test.xml.ref: updated
- 02:34 PM Revision 13944: inputs/FIA/TREE/test.xml.ref: updated
- 02:29 PM Revision 13943: inputs/FIA/REF_RESEARCH_STATION/test.xml.ref: updated
- 02:13 PM Revision 13942: bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new TWiki/ location
- 02:12 PM Revision 13941: bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: updated to new VegBIEN/ location
- 12:14 PM Revision 13940: fix: inputs/bien2_traits/TraitObservation/map.csv: mapped VisitingDate to eventDate
- 12:05 PM Revision 13939: fix: inputs/Madidi/LocationObservation/postprocess.sql: populated missing eventDate from PlotInventoryName (authorEventCode)
- 09:40 AM Revision 13938: schemas/util.sql: mk_set_comment(table_ regclass, comment text): use new util.mk_set_comment(text, text)
- 09:38 AM Revision 13937: schemas/util.sql: added mk_set_comment(col col_ref, comment text)
- 09:37 AM Revision 13936: schemas/util.sql: added sql(col_ref)
- 09:35 AM Revision 13935: schemas/util.sql: added mk_set_comment(on_ text, comment text)
- 07:53 AM Revision 13934: schemas/util.sql: added comment(col_ref)
- 07:50 AM Revision 13933: schemas/util.sql: added col_num(col_ref)
- 07:03 AM Revision 13932: removed no longer needed inputs/.TNRS/grants.sql, since the grants in schema.sql are now being run
- 07:03 AM Revision 13931: bugfix: inputs/input.Makefile: sql/install: schema.sql should not be passed through pg_dump_limit because it contains GRANT statements that need to be run
- 06:22 AM Revision 13930: schemas/public_.sql: iucn_red_list_view: documented how to regenerate iucn_red_list from this
07/08/2014
- 04:17 PM Revision 13929: exports/2014-6-4.Iara_Lacher.reserve_prioritization.csv.run: documented runtime (7.5 min) and rowcount (3.5 million)
- 03:23 PM Revision 13928: bugfix: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: fixed slowdown in materializing the view, which was caused by source__observation_type(), by replacing this with a LEFT JOIN as was done for is_threatened_iucn
07/07/2014
- 07:33 AM Revision 13927: schemas/util.sql: in_south_america(): optimized by using BETWEEN instead of util.contained_within__no_dateline(). this reduces the filter time for 2014-6-4.Iara_Lacher.reserve_prioritization by about 10%.
- 07:26 AM Revision 13926: fix: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: is_threatened_iucn: generate this much faster by doing a LEFT JOIN, which can reuse the same in-memory hash index for every row
- 07:04 AM Revision 13925: bugfix: schemas/public_.sql: wrapper functions can't use util-schema types because these will cause the wrapper function to be cascadingly dropped when the util schema is reinstalled
- 06:55 AM Revision 13924: bugfix: in_south_america(): must use util.contained_within__no_dateline() instead of util.contained_within_approx() to ensure that the more accurate geometry logic is used
- 06:51 AM Revision 13923: bugfix: schemas/public_.sql: source__observation_type(): don't display NOTICEs about the search_path, because this function will be called millions of times
- 06:48 AM Revision 13922: bugfix: schemas/public_.sql: in_iucn_red_list(): don't display NOTICEs about the search_path, because this function will be called millions of times
- 06:43 AM Revision 13921: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added is_threatened_iucn
- 06:37 AM Revision 13920: schemas/public_.sql: added in_iucn_red_list()
- 06:36 AM Revision 13919: schemas/public_.sql: added iucn_red_list, materialized from iucn_red_list_view
- 06:22 AM Revision 13918: bugfix: schemas/vegbien.sql: iucn_red_list_view: need to include only names with an accepted name, and distinctify on the accepted names
- 06:17 AM Revision 13917: schemas/public_.sql: added iucn_red_list
- 06:14 AM Revision 13916: schemas/util.sql: materialize_view(): use util.copy() instead of util.materialize_query() so that all view metadata is transferred
- 06:02 AM Revision 13915: schemas/public_.sql: iucn_red_list: renamed to iucn_red_list_view since this will be materialized
- 06:01 AM Revision 13914: schemas/public_.sql: added iucn_red_list
- 05:44 AM Revision 13913: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added collection_type (using source__observation_type())
- 05:36 AM Revision 13912: schemas/public_.sql: added source__observation_type()
- 05:04 AM Revision 13911: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added accepted_subspecies, and removed accepted_taxon_name_with_author which had been substituting for it
- 04:50 AM Revision 13910: schemas/public_.sql: added subspecies() wrapper
- 04:38 AM Revision 13909: schemas/util.sql: added subspecies()
- 04:10 AM Revision 13908: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: include only coordinates in South America
- 03:55 AM Revision 13907: schemas/util.sql: added south_america(), in_south_america()
- 03:37 AM Revision 13906: fix: schemas/util.sql: OPERATOR ~@(geocoord, postgis.geometry): renamed to @ because this is not an approximate comparison for geometry
- 03:35 AM Revision 13905: fix: schemas/util.sql: contained_within_approx(geocoord, postgis.geometry): renamed to contained_within__no_dateline(__) because this is not an approximate comparison for geometry
- 03:30 AM Revision 13904: schemas/util.sql: lat_long_in_new_world(): renamed to just in_new_world() because the lat/long is implied by the param type
- 03:08 AM Revision 13903: schemas/util.sql: lat_long_in_new_world(): take a geocoord param instead of separate lat/long params
- 03:00 AM Revision 13902: backups/TNRS.backup.md5: updated
07/06/2014
- 11:08 PM Revision 13901: schemas/util.sql: added contained_within_approx(geocoord, geometry) and corresponding OPERATOR ~@(geocoord, geometry)
- 10:39 PM Revision 13900: schemas/util.sql: added OPERATOR ~@(geocoord, geography)
- 10:23 PM Revision 13899: schemas/util.sql: lat_long_in_new_world(): use new contained_within_approx(geocoord, geography)
- 10:17 PM Revision 13898: schemas/util.sql: added contained_within_approx(geocoord, postgis.geography), which enables specifying just `(lat, long)` without the ::util.geocoord type specifier
- 04:04 PM Revision 13897: schemas/util.sql: OPERATOR @(postgis.geography, postgis.geography): renamed to ~@ because it's approximate
- 03:52 PM Revision 13896: schemas/util.sql: contained_within(): renamed to contained_within_approx() because the latitude lines of geography type bounding boxes bulge outward, creating false positives above and below the bounding box
- 03:50 PM Revision 13895: schemas/util.sql: added contained_within__no_dateline(geometry, geometry) and corresponding operator @
- 02:51 PM Revision 13894: schemas/util.sql: contained_within(): renamed to contained_within_approx() because the latitude lines of geography type bounding boxes bulge outward, creating false positives above and below the bounding box
- 05:49 AM Revision 13893: schemas/util.sql: geometry(geocoord): documented that it is not possible to create a cast for this, as a bug in pg_dump prevents the cast from being exported, even when no export filters are applied
07/05/2014
- 12:40 PM Revision 13892: schemas/util.sql: point(geocoord): renamed to geometry(geocoord) since this is now a cast
- 12:03 PM Revision 13891: schemas/util.sql: point(): return geometry instead of geography to support using points with geometry arithmetic
- 11:24 AM Revision 13890: schemas/util.sql: point(): take a single util.geocoord param instead of separate lat/long
- 10:45 AM Revision 13889: schemas/util.sql: added geocoord type
07/04/2014
- 08:29 PM Revision 13888: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: include only *georeferenced* occurrences (lat/long NOT NULL)
- 08:25 PM Revision 13887: schemas/util.sql: bounding_box(): use bounding_box__no_dateline() to construct the postgis.geometry object
- 08:23 PM Revision 13886: schemas/util.sql: added bounding_box__no_dateline(), which is more accurate than util.bounding_box() (latitude lines will be straight), but geocoordinate wraparound is not supported
- 08:14 PM Revision 13885: schemas/public_.sql: 2014-6-4.Iara_Lacher.reserve_prioritization: added functional traits that we have 1st-class columns for (dbh_cm, height_m)
- 12:29 PM Revision 13884: added exports/2014-6-4.Iara_Lacher.reserve_prioritization.csv.run
- 11:12 AM Revision 13883: inputs/publishable datasources.xlsx: updated
- 08:45 AM Revision 13882: inputs/publishable datasources.xlsx: updated
07/03/2014
- 08:13 PM Revision 13881: inputs/.TNRS/schema.sql: matchedFamily: just use Name_matched_accepted_family, because TNRS has now been reloaded so that the names that were missing this have it populated
- 09:59 AM Revision 13880: planning/meetings/BIEN conference call availability.xlsx: updated
- 12:34 AM Revision 13879: inputs/.TNRS/schema.sql: taxon_match: taxon_match__valid_match: replaced with taxon_best_match__valid_match, which also applies taxon_best_match's filters, since taxon_match is now accessed through taxon_best_match
- 12:17 AM Revision 13878: fix: inputs/.TNRS/schema.sql: MatchedTaxon: use taxon_best_match instead of taxon_match because this should provide only one match per taxon
07/02/2014
- 02:17 AM Revision 13877: bugfix: /README.TXT: Mac settings backup: backup to jupiter: need to exclude ~/software/**/.svn/ because these are different on jupiter
- 02:00 AM Revision 13876: /README.TXT: Mac settings backup: backup to jupiter: removed no longer applicable exclude of /VirtualBox VMs/Ubuntu/Ubuntu.vdi
Also available in: Atom