Project

General

Profile

Activity

From 05/06/2014 to 06/04/2014

06/04/2014

03:01 PM Revision 13630: fix: lib/tnrs.py: max_names: lowered to 50 because the dev TNRS server is now always crashing with an Internal Server Error when scrubbing 500 names at a time (https://github.com/iPlantCollaborativeOpenSource/TNRS/issues/7)
Aaron Marcuse-Kubitza
11:41 AM Revision 13629: web/.htaccess: auto-detect dotpath in query string: also auto-prepend & for query strings appended to a username-based prefix, so that these don't need to have & manually prepended to make them appendable
Aaron Marcuse-Kubitza
11:14 AM Revision 13628: web/.htaccess: auto-detect dotpath in query string: also auto-detect dotpath for query strings appended to a username-based prefix, so that these don't need to have . prepended to make them appendable
Aaron Marcuse-Kubitza
02:25 AM Revision 13627: web/username_prefix.php: no prefix: use $_SERVER["SCRIPT_URL"] instead of rtrimming $_SERVER["REQUEST_URI"]
Aaron Marcuse-Kubitza
02:24 AM Revision 13626: web/username_prefix.php: has prefix: use same expr whether or not it has a query string, because the with-query-string expr works for both cases
Aaron Marcuse-Kubitza
02:20 AM Revision 13625: web/username_prefix.php: use if statements instead of `? :` for clarity
Aaron Marcuse-Kubitza
02:10 AM Revision 13624: web/username_prefix.php: use $subpath instead of $_SERVER["PHP_AUTH_USER"] for clarity
Aaron Marcuse-Kubitza
02:09 AM Revision 13623: bugfix: **/.htaccess: redirects with fragment: qsappend does not support fragment, so append it separately
Aaron Marcuse-Kubitza
01:59 AM Revision 13622: web/.htaccess, username_prefix.php: username-based prefix subpaths: support dotpath in query string (which will be appended verbatim to the prefix). use "__?@__" instead of "__?__&" to signal username-based prefix, so that the query string does not end in a dangling & .
Aaron Marcuse-Kubitza
01:53 AM Revision 13621: bugfix: inputs/.TNRS/.htaccess: qsappend does not support fragment, so append it separately
Aaron Marcuse-Kubitza
01:06 AM Revision 13620: bugfix: web/.htaccess: auto-detect dotpath in query string: don't exclude dotpaths that come before a normal query string (&k=v...)
Aaron Marcuse-Kubitza
01:05 AM Revision 13619: web/.htaccess: auto-detect dotpath in query string: documented what each RewriteCond does
Aaron Marcuse-Kubitza
01:03 AM Revision 13618: web/.htaccess: auto-detect dotpath in query string: use %{QUERY_STRING} instead of %0 for clarity
Aaron Marcuse-Kubitza
12:48 AM Revision 13617: web/.htaccess, web/username_prefix.php: username-based prefix subpaths: support query strings in addition to paths
Aaron Marcuse-Kubitza
12:18 AM Revision 13616: web/index.php: removed no longer needed redirection of username-based prefix subpaths, which is now handled by web/.htaccess
Aaron Marcuse-Kubitza
12:16 AM Revision 13615: web/.htaccess: handle username-based prefix subpaths (of the form "subpath@url?") for all paths rather than just for the root (which were handled by index.php)
Aaron Marcuse-Kubitza
12:11 AM Revision 13614: added web/username_prefix.php, for use with username-based prefix redirects
Aaron Marcuse-Kubitza

06/03/2014

11:13 PM Revision 13613: web/index.php: subpath@host? redirects: also support multiple @ and nested . in prefix: a@b.c@host? -> host?b.c.a
Aaron Marcuse-Kubitza
11:01 PM Revision 13612: web/index.php: support prefix redirect of the form subpath@host? , which is better than subpath.host because case is preserved and special chars allowed. (must require trailing ? for redirect, to avoid needing login to view the page itself.)
Aaron Marcuse-Kubitza
10:47 PM Revision 13611: bugfix: web/index.php: full directory index: appending query string: need to use $_SERVER["QUERY_STRING"], not $_SERVER["HTTP_AUTHORIZATION"] for this
Aaron Marcuse-Kubitza
11:36 AM Revision 13610: schemas/public_.sql: sync_*(): renamed to *_modify() to facilitate finding these functions when modifying the corresponding view (using the new naming convention for a view's on-modify function)
Aaron Marcuse-Kubitza
10:36 AM Revision 13609: bugfix: inputs/.TNRS/schema.sql: MatchedTaxon_modify(): updated to include taxon_scrub derived fields
Aaron Marcuse-Kubitza
10:25 AM Revision 13608: bugfix: schemas/util.sql: mk_drop_from_create(): need to match first rather than last CREATE
Aaron Marcuse-Kubitza
10:21 AM Revision 13607: inputs/.TNRS/schema.sql: *_modify(): allow running without a view_query, as recreate_view() now supports this
Aaron Marcuse-Kubitza
09:52 AM Revision 13606: schemas/util.sql: recreate_view(): support omitting the view_query if the view has already been modified (eg. for public.*_view, which allow changing the view as a separate step)
Aaron Marcuse-Kubitza
08:56 AM Revision 13605: fix: schemas/public_.sql: sync_*(): use util.copy() instead of CREATE TABLE AS so that table and column comments are also copied. this avoids the need to separately add the same comments to the view *and* its materialized table.
Aaron Marcuse-Kubitza
08:52 AM Revision 13604: bugfix: schemas/util.sql: recreate(): need to handle case where util.mk_drop_from_create() is NULL
Aaron Marcuse-Kubitza
08:45 AM Revision 13603: bugfix: schemas/util.sql: mk_drop_from_create(): only match CREATE if no custom DROP came before it
Aaron Marcuse-Kubitza
08:29 AM Revision 13602: bugfix: schemas/public_.sql: sync_geoscrub_input_to_view(): `CREATE TABLE geoscrub_input AS __`: needs `LIMIT 0`
Aaron Marcuse-Kubitza
08:19 AM Revision 13601: fix: schemas/util.sql: explain2notice_msg_if_can(): also need to catch invalid_cursor_definition ("cannot open multi-query plan as cursor")
Aaron Marcuse-Kubitza
08:15 AM Revision 13600: schemas/public_.sql: sync_analytical_stem_to_view(): removed DROP TABLE IF EXISTS because this is now done automatically by util.recreate()
Aaron Marcuse-Kubitza
07:39 AM Revision 13599: schemas/util.sql: added copy()
Aaron Marcuse-Kubitza
07:36 AM Revision 13598: schemas/util.sql: added copy_data()
Aaron Marcuse-Kubitza

06/02/2014

04:24 PM Revision 13597: fix: lib/tnrs.py: Constrain by Source: turn it on so that the download settings reflect what TNRS actually used, while this is broken
Aaron Marcuse-Kubitza
06:19 AM Revision 13596: fix: lib/tnrs.py: max_names: reduced back to 500 because even 5000 crashes the dev TNRS server
Aaron Marcuse-Kubitza
05:52 AM Revision 13595: lib/tnrs.py: max_names: reduced to 5000 because 100,000 causes an internal server error
Aaron Marcuse-Kubitza
05:25 AM Revision 13594: bugfix: /README.TXT: Full database import: To run TNRS: to rescrub all names: also need to re-create public-schema views that were cascadingly deleted
Aaron Marcuse-Kubitza
05:23 AM Revision 13593: /README.TXT: Full database import: To run TNRS: added steps to rescrub all names
Aaron Marcuse-Kubitza
04:59 AM Revision 13592: backups/TNRS.backup.md5: updated
Aaron Marcuse-Kubitza
04:50 AM Revision 13591: lib/tnrs.py: switched to downloading all matches per name, as is needed to implement #917. note that this will break the parts of the schema that use the tnrs table, until Brad's match-picking algorithm can be implemented, but this tradeoff is necessary to be able to begin scrubbing sooner (Martha; wiki.vegpath.org/2014-05-29_conference_call#TNRS)
Aaron Marcuse-Kubitza
04:35 AM Revision 13590: schemas/vegbien.sql: tnrs_input_name: don't scrub accepted names, as using multiple matches per name no longer provides a single accepted name to scrub. instead, the Accepted_* fields can be whitespace-split to generate the same columns that would have been generated by the scrubbing (and without the overhead of the extra TNRS call).
Aaron Marcuse-Kubitza
04:27 AM Revision 13589: fix: inputs/.TNRS/schema.sql: added back index on Name_submitted, which is needed for tnrs_input_name to work properly (now that there is no automatic index created by a unique constraint)
Aaron Marcuse-Kubitza
04:08 AM Revision 13588: schemas/vegbien.sql: tnrs_input_name: don't scrub accepted names, as using multiple matches per name no longer provides a single accepted name to scrub. instead, the Accepted_* fields can be whitespace-split to generate the same columns that would have been generated by the scrubbing (and without the overhead of the extra TNRS call).
Aaron Marcuse-Kubitza
03:43 AM Revision 13587: fix: inputs/.TNRS/schema.sql: tnrs: removed unique constraint on Name_submitted, Name_matched because there can be more than one match with the same Name_matched (but different accepted names, etc.)
Aaron Marcuse-Kubitza

06/01/2014

09:00 PM Revision 13586: fix: inputs/.TNRS/schema.sql: tnrs.tnrs__valid_match index: made it non-unique to allow multiple matches per name, as is needed to implement #917
Aaron Marcuse-Kubitza
05:00 AM Revision 13585: bugfix: inputs/.TNRS/schema.sql: tnrs__match_num__fill(): only fill if not set, to support case where tnrs is being restored from a .sql file (where match_num is already set)
Aaron Marcuse-Kubitza
04:36 AM Revision 13584: inputs/.TNRS/schema.sql: tnrs: documented runtime to add a constraint (3 min)
Aaron Marcuse-Kubitza
04:35 AM Revision 13583: inputs/.TNRS/schema.sql: unique constraint on Name_submitted: added Name_matched to allow multiple matches per name, as is needed to implement #917
Aaron Marcuse-Kubitza
03:44 AM Revision 13582: inputs/.TNRS/schema.sql: tnrs: documented how to populate a new column
Aaron Marcuse-Kubitza
03:41 AM Revision 13581: inputs/.TNRS/schema.sql: tnrs: pkey: use match_num instead of Name_number to allow multiple matches per name, as is needed to implement #917
Aaron Marcuse-Kubitza

05/31/2014

10:31 PM Revision 13580: inputs/.TNRS/schema.sql: tnrs.match_num: made it NOT NULL now that it's populated
Aaron Marcuse-Kubitza
10:28 PM Revision 13579: inputs/.TNRS/schema.sql: tnrs: populate match_num
Aaron Marcuse-Kubitza
10:25 PM Revision 13578: inputs/.TNRS/schema.sql: tnrs: populate match_num
Aaron Marcuse-Kubitza
09:50 PM Revision 13577: inputs/.TNRS/schema.sql: tnrs: documented how to add and remove columns
Aaron Marcuse-Kubitza
08:58 PM Revision 13576: inputs/.TNRS/schema.sql: made COMMENTs start on their own line, using the steps at wiki.vegpath.org/Postgres_queries#make-COMMENTs-start-on-their-own-line
Aaron Marcuse-Kubitza
08:58 PM Revision 13575: inputs/.TNRS/schema.sql: made COMMENTs start on their own line, using the steps at wiki.vegpath.org/Postgres_queries#make-COMMENTs-start-on-their-own-line
Aaron Marcuse-Kubitza
08:11 PM Revision 13574: inputs/test_taxonomic_names/_scrub/*: updated to TNRS schema
Aaron Marcuse-Kubitza
08:10 PM Revision 13573: inputs/.TNRS/schema.sql: tnrs: added match_num
Aaron Marcuse-Kubitza
08:06 PM Revision 13572: inputs/.TNRS/data.sql.run: refresh(): documented runtime (1 min)
Aaron Marcuse-Kubitza
08:03 PM Revision 13571: schemas/Makefile: added back rename/%, which *is* used by `inputs/.TNRS/data.sql.run refresh`. updated it to use schema bundles.
Aaron Marcuse-Kubitza
06:44 PM Revision 13570: inputs/.TNRS/schema.sql: added tnrs__match_num__next()
Aaron Marcuse-Kubitza
06:29 PM Revision 13569: inputs/.TNRS/schema.sql: added tnrs__batch_begin() trigger to populate the match_num (match sort order)
Aaron Marcuse-Kubitza
05:06 AM Revision 13568: exports/2014-3-11.Jeff_Ott.climatic_range_determinants.csv.run: documented export_() runtime (11 min) and rowcount (11 million matching the filter criteria)
Aaron Marcuse-Kubitza

05/30/2014

06:34 PM Revision 13567: inputs/.TNRS/schema.sql: added tnrs__batch_begin() trigger to populate the match_num (match sort order)
Aaron Marcuse-Kubitza
06:22 PM Revision 13566: schemas/util.sql: added seq__reset()
Aaron Marcuse-Kubitza
06:15 PM Revision 13565: schemas/util.sql: added seq__create()
Aaron Marcuse-Kubitza
06:11 PM Revision 13564: fix: schemas/util.sql: try_cast(), is_castable(): also catch invalid_schema_name, thrown by `'pg_temp.__'::regclass`
Aaron Marcuse-Kubitza
02:49 PM Revision 13563: backups/TNRS.backup.md5: updated
Aaron Marcuse-Kubitza
07:50 AM Revision 13562: lib/tnrs.py: max_names: increased to 100000 because the dev server can handle more names (no simultaneous users), as decided in the conference call (wiki.vegpath.org/2014-05-29_conference_call#TNRS)
Aaron Marcuse-Kubitza
07:42 AM Revision 13561: schemas/vegbien.ERD.mwb: regenerated exports
Aaron Marcuse-Kubitza
07:39 AM Revision 13560: schemas/vegbien.ERD.mwb: re-updated to schemas/vegbien.my.sql, which now recognizes the broken tables. fixed sync issues. vegbien.ERD.mwb is now fully in sync with vegbien.my.sql.
Aaron Marcuse-Kubitza
07:37 AM Revision 13559: fix: lib/PostgreSQL-MySQL.csv: need to replace "double precision" with "double" to work with MySQL Workbench 5.2.47
Aaron Marcuse-Kubitza
06:58 AM Revision 13558: schemas/vegbien.ERD.mwb: updated to schemas/vegbien.my.sql. some tables weren't recognized (likely due to bugs in MySQL Workbench 5.2.47), and have been left as-is (unsynced). note that downgrading to 5.2.35 is not an option, because that is fatally broken by a system upgrade.
Aaron Marcuse-Kubitza
05:58 AM Revision 13557: fix: schemas/vegbien.ERD.mwb: use schemas/vegbien.my.sql instead of schemas/vegbien.my.sql.changes.sql as the sync source
Aaron Marcuse-Kubitza
05:27 AM Revision 13556: schemas/vegbien.ERD.mwb: switched back to MySQL Workbench 6.1.6 version, which also works with MySQL Workbench 5.2.47
Aaron Marcuse-Kubitza
05:17 AM Revision 13555: schemas/vegbien.ERD.mwb: restored version for MySQL Workbench 5.2.35 (undid r13549), as 6.1.6 has bugs in the DDL file sync
Aaron Marcuse-Kubitza
05:05 AM Revision 13554: schemas/vegbien.ERD.mwb: renamed to vegbien.ERD.MySQL_Workbench_6.1.6.mwb to differentiate the versions for different versions of MySQL Workbench
Aaron Marcuse-Kubitza
04:49 AM Revision 13553: removed no longer used config/VirtualBox_VMs/VegCore ERD/. use the Ubuntu 14.04 VM instead, which now has the VegCore ERD setup.
Aaron Marcuse-Kubitza
04:48 AM Revision 13552: config/VirtualBox_VMs/Ubuntu 14.04/Ubuntu 14.04.vbox: added VegCore ERD setup
Aaron Marcuse-Kubitza
03:30 AM Revision 13551: schemas/: svn:ignore *.changes.sql, needed for MySQL Workbench 6.1.6
Aaron Marcuse-Kubitza
03:30 AM Revision 13550: schemas/vegbien.ERD.mwb: updated
Aaron Marcuse-Kubitza
03:16 AM Revision 13549: schemas/vegbien.ERD.mwb: updated layout for MySQL Workbench 6.1.6, which uses a different line spacing
Aaron Marcuse-Kubitza

05/29/2014

11:53 AM Revision 13548: lib/tnrs.py: commented out the value of max_names that is not active, for clarity
Aaron Marcuse-Kubitza

05/28/2014

03:54 PM Revision 13547: config/VirtualBox_VMs/vegbiendev/vegbiendev.vbox: updated: merged post-bootloader installation steps into one snapshot now that eth1 addition is successful
Aaron Marcuse-Kubitza
03:49 PM Revision 13546: config/VirtualBox_VMs/vegbiendev/vegbiendev.vbox: updated, which adds eth1
Aaron Marcuse-Kubitza
03:47 PM Revision 13545: config/VirtualBox_VMs/Ubuntu 12.04/Ubuntu 12.04.vbox: updated
Aaron Marcuse-Kubitza

05/27/2014

11:12 PM Revision 13544: lib/tnrs.py: sources: updated to list/sort order in issue #917
Aaron Marcuse-Kubitza
10:39 PM Revision 13543: added exports/2014-3-11.Jeff_Ott.climatic_range_determinants.csv.run
Aaron Marcuse-Kubitza
10:38 PM Revision 13542: schemas/public_.sql: added 2014-3-11.Jeff_Ott.climatic_range_determinants
Aaron Marcuse-Kubitza
10:17 PM Revision 13541: schemas/public_.sql: analytical_stem_view: added scrubbed_taxon_name_with_author, needed by Jeff Ott's analysis (wiki.vegpath.org/Data_requests)
Aaron Marcuse-Kubitza
10:13 PM Revision 13540: inputs/.TNRS/schema.sql: taxon_scrub.scrubbed_unique_taxon_name.*: added scrubbed_taxon_name_with_author, needed by Jeff Ott's analysis (wiki.vegpath.org/Data_requests)
Aaron Marcuse-Kubitza
10:09 PM Revision 13539: schemas/public_.sql: added scrubbed_specific_epithet, scrubbed_species_binomial, which are needed by Jeff Ott's analysis (wiki.vegpath.org/Data_requests)
Aaron Marcuse-Kubitza
10:08 PM Revision 13538: fix: schemas/public_.sql: sync_analytical_stem_to_view(): removed fkey to source.shortname because this prevents reloading individual datasources
Aaron Marcuse-Kubitza
09:19 PM Revision 13537: added downloads/
Aaron Marcuse-Kubitza
03:27 AM Revision 13536: schemas/public_.sql: added scrubbed_specific_epithet, scrubbed_species_binomial, which are needed by Jeff Ott's analysis (wiki.vegpath.org/Data_requests)
Aaron Marcuse-Kubitza
03:21 AM Revision 13535: fix: schemas/util.sql: mk_drop_from_create(): also support CREATE queries that include the SELECT statement on the same line as the CREATE
Aaron Marcuse-Kubitza
12:33 AM Revision 13534: schemas/public_.sql: analytical_stem_view: scrubbed_morphospecies_binomial: use new taxon_scrub.scrubbed_morphospecies_binomial
Aaron Marcuse-Kubitza
12:28 AM Revision 13533: inputs/.TNRS/schema.sql: taxon_scrub: added scrubbed_morphospecies_binomial, analogous to accepted_morphospecies_binomial for scrubbed_*
Aaron Marcuse-Kubitza
12:13 AM Revision 13532: inputs/.TNRS/schema.sql: taxon_scrub: added scrubbed_morphospecies_binomial, analogous to accepted_morphospecies_binomial for scrubbed_*
Aaron Marcuse-Kubitza

05/26/2014

11:54 PM Revision 13531: inputs/.TNRS/schema.sql: taxon_scrub: documented how to modify it
Aaron Marcuse-Kubitza
11:51 PM Revision 13530: inputs/.TNRS/schema.sql: added taxon_scrub_modify()
Aaron Marcuse-Kubitza
11:41 PM Revision 13529: schemas/util.sql: create_if_not_exists(): print message if already exists, so the function doesn't inexplicably appear not to have run at all
Aaron Marcuse-Kubitza
11:20 PM Revision 13528: inputs/.TNRS/schema.sql: added taxon_scrub_modify()
Aaron Marcuse-Kubitza

05/23/2014

06:17 PM Revision 13527: inputs/.TNRS/schema.sql: MatchedTaxon_modify(): use simpler util.recreate_view()
Aaron Marcuse-Kubitza
06:15 PM Revision 13526: inputs/.TNRS/schema.sql: MatchedTaxon_modify(): documented usage
Aaron Marcuse-Kubitza
06:09 PM Revision 13525: schemas/util.sql: added recreate_view(), a special case of util.recreate()
Aaron Marcuse-Kubitza
04:24 PM Revision 13524: fix: schemas/util.sql: recreate(): usage: use `schema` instead of `schemas`
Aaron Marcuse-Kubitza
04:20 PM Revision 13523: config/VirtualBox_VMs/vegbiendev/vegbiendev.vbox: updated
Aaron Marcuse-Kubitza
03:57 PM Revision 13522: bugfix: schemas/public_.sql: _plots_20_tnrs_names: verbatim_name_with_author: use taxonverbatim.taxonomicname rather than taxonlabel.taxonomicname
Aaron Marcuse-Kubitza
02:16 AM Revision 13521: config/VirtualBox_VMs/Ubuntu */*.vbox: updated
Aaron Marcuse-Kubitza

05/22/2014

04:20 PM Revision 13520: bugfix: config/VirtualBox_VMs/: switched from symlinks to hard links, because svn does not follow symlinks
Aaron Marcuse-Kubitza
02:07 PM Revision 13519: added config/VirtualBox_VMs/, containing the .vbox settings and password.txt (a non-empty password is needed for some system commands)
Aaron Marcuse-Kubitza

05/21/2014

07:30 PM Revision 13518: inputs/.TNRS/schema.sql: MatchedTaxon_modify(): removed no longer needed DROP VIEW statement
Aaron Marcuse-Kubitza
07:30 PM Revision 13517: schemas/util.sql: recreate(): perform the correct DROP VIEW in the function itself so that the caller does not have to worry about forming it properly
Aaron Marcuse-Kubitza
07:19 PM Revision 13516: bugfix: schemas/util.sql: mk_drop_from_create(): added `DROP`
Aaron Marcuse-Kubitza
07:18 PM Revision 13515: schemas/util.sql: added mk_drop_from_create()
Aaron Marcuse-Kubitza
06:52 PM Revision 13514: schemas/util.sql: added regexp_match()
Aaron Marcuse-Kubitza
05:58 PM Revision 13513: planning/meetings/BIEN conference call availability.xlsx: updated
Aaron Marcuse-Kubitza
05:50 PM Revision 13512: fix: schemas/util.sql: force_recreate(): renamed to just recreate(), because "force" normally implies that things will be deleted, which this function does not do
Aaron Marcuse-Kubitza
05:49 PM Revision 13511: fix: schemas/public_.sql: _plots_20_tnrs_names: fixed order of tnrs_taxonomic_status, tnrs_accepted_name_family
Aaron Marcuse-Kubitza
04:45 PM Revision 13510: schemas/public_.sql: _plots_20_tnrs_names: added tnrs_taxonomic_status, as requested in issue #915
Aaron Marcuse-Kubitza
04:34 PM Revision 13509: schemas/public_.sql: analytical_stem_view: taxonomic_status: don't filter with map_taxonomic_status() anymore since this is now done by "TNRS"."MatchedTaxon"
Aaron Marcuse-Kubitza
04:25 PM Revision 13508: fix: inputs/.TNRS/schema.sql: MatchedTaxon.taxonomicStatus: filter using map_taxonomic_status() so that the corrected value is available in the normalized DB, not just analytical_stem
Aaron Marcuse-Kubitza
04:05 PM Revision 13507: inputs/.TNRS/schema.sql: MatchedTaxon: to modify: use new MatchedTaxon_modify(), which eliminates the work of putting together the dependent views
Aaron Marcuse-Kubitza
03:53 PM Revision 13506: inputs/.TNRS/schema.sql: added MatchedTaxon_modify()
Aaron Marcuse-Kubitza
03:44 PM Revision 13505: schemas/util.sql: show_create_view(): use new mk_set_relation_metadata()
Aaron Marcuse-Kubitza
03:41 PM Revision 13504: schemas/util.sql: added mk_set_relation_metadata()
Aaron Marcuse-Kubitza
04:13 AM Revision 13503: bugfix: inputs/.TNRS/schema.sql: map_taxonomic_status(): need to use accepted name instead of scrubbed name (which also includes no-opinion names), as described at http://wiki.vegpath.org/2013-11-14_conference_call#taxonomic-fields. this used to be the accepted name, but got switched when the concatenated name was also used to store the matched name for no-opinion names.
Aaron Marcuse-Kubitza
01:50 AM Revision 13502: schemas/public_.sql: _plots_20_tnrs_names: use accepted_morphospecies_binomial, which is now available directly in the TNRS results, instead of just the morphospecies_suffix, as requested in issue #915
Aaron Marcuse-Kubitza
01:27 AM Revision 13501: inputs/.TNRS/schema.sql: MatchedTaxon: documented how to modify it (using util.force_recreate())
Aaron Marcuse-Kubitza
01:23 AM Revision 13500: fix: lib/PostgreSQL-MySQL.csv: also remove left-behind lines such as `$$);`
Aaron Marcuse-Kubitza
12:59 AM Revision 13499: schemas/util.sql: force_recreate(): documented usage, which is somewhat complex and has several important parts
Aaron Marcuse-Kubitza

05/20/2014

05:46 PM Revision 13498: inputs/.TNRS/schema.sql: MatchedTaxon, etc.: added accepted_morphospecies_binomial derived field
Aaron Marcuse-Kubitza
05:40 PM Revision 13497: bugfix: schemas/util.sql: restore_views(): need to specify view name for a manual existence check, in case the view definition becomes invalid, which would produce a nonstandard (uncatchable) exception
Aaron Marcuse-Kubitza
05:33 PM Revision 13496: bugfix: schemas/util.sql: explain(): don't display any log messages printed by IMMUTABLE functions in the query, which EXPLAIN apparently runs
Aaron Marcuse-Kubitza
05:21 PM Revision 13495: schemas/util.sql: create_if_not_exists(): added relation param, which can be used to ensure that a standard exception is always generated if the relation exists, even if the table definition would be invalid (which generates a variety of exceptions)
Aaron Marcuse-Kubitza
02:39 PM Revision 13494: /README.TXT: to back up the vegbiendev databases: added step to review diff before performing backup, in case the backup scripts break and try to synchronize things incorrectly
Aaron Marcuse-Kubitza

05/19/2014

09:08 PM Revision 13493: bugfix: schemas/util.sql: try_cast(), is_castable(): also need to handle syntax_error_or_access_rule_violation, which is thrown by ::regclass
Aaron Marcuse-Kubitza
08:52 PM Revision 13492: schemas/util.sql: added relation_exists()
Aaron Marcuse-Kubitza
08:27 PM Revision 13491: schemas/util.sql: save_drop_views()/restore_views(): store the view path in addition to the definition so it can be used in restoring
Aaron Marcuse-Kubitza
08:24 PM Revision 13490: bugfix: schemas/util.sql: in_reverse(): can't use `SELECT *` for composite types because this expands them
Aaron Marcuse-Kubitza
08:21 PM Revision 13489: bugfix: schemas/util.sql: array_reverse(): can't use `SELECT *` for composite types because this expands them
Aaron Marcuse-Kubitza
07:50 PM Revision 13488: schemas/util.sql: save_drop_views()/restore_views(): use opaque return type so it can be changed without changing the local var type in functions that use this
Aaron Marcuse-Kubitza
07:22 PM Revision 13487: schemas/util.sql: force_recreate(): recreate_users_cmds: renamed to restore_views_info since this is now whatever is needed by util.restore_views()
Aaron Marcuse-Kubitza
07:16 PM Revision 13486: schemas/util.sql: added restore_views() and use it in force_recreate()
Aaron Marcuse-Kubitza
02:07 AM Revision 13485: bugfix: schemas/util.sql: save_drop_views(): views must be *dropped* in reverse dependency order, but *returned* in forward dependency order
Aaron Marcuse-Kubitza
01:36 AM Revision 13484: bugfix: schemas/util.sql: force_recreate(): dependent views need to be re-created in *forward* dependency order. this is done by reversing the dependencies only for save_drop_views().
Aaron Marcuse-Kubitza
12:48 AM Revision 13483: bugfix: schemas/util.sql: mk_set_comment(): need to use VIEW instead of TABLE for views
Aaron Marcuse-Kubitza
12:29 AM Revision 13482: bugfix: schemas/util.sql: show_create_view(): also need to include view comment
Aaron Marcuse-Kubitza
12:24 AM Revision 13481: fix: schemas/util.sql: mk_set_comment(): added ;
Aaron Marcuse-Kubitza
12:20 AM Revision 13480: fix: schemas/util.sql: show_set_comment(): don't display a COMMENT ON statement if no comment, because this will be appended to table defs, etc. and would create clutter
Aaron Marcuse-Kubitza
12:16 AM Revision 13479: bugfix: schemas/util.sql: mk_set_comment(): need to handle NULL comment properly
Aaron Marcuse-Kubitza
12:06 AM Revision 13478: schemas/util.sql: added show_set_comment(regclass)
Aaron Marcuse-Kubitza

05/18/2014

11:57 PM Revision 13477: schemas/util.sql: set_comment(): use util.mk_set_comment()
Aaron Marcuse-Kubitza
11:48 PM Revision 13476: schemas/Makefile: added vegbien/install, vegbien/uninstall which also handle TNRS
Aaron Marcuse-Kubitza

05/17/2014

06:26 PM Revision 13475: fix: schemas/Makefile: vegbien.sql: also remake inputs/.TNRS/schema.sql since its contents change along with vegbien.sql
Aaron Marcuse-Kubitza
06:22 PM Revision 13474: schemas/util.sql: added mk_set_comment()
Aaron Marcuse-Kubitza
06:22 PM Revision 13473: bugfix: lib/runscripts/util.run: $is_runscript: unexport so don't pass it to invoked scripts
Aaron Marcuse-Kubitza
06:01 PM Revision 13472: backups/pg_snapshot: documented runtime for initial backup (4 h)
Aaron Marcuse-Kubitza
05:36 PM Revision 13471: bugfix: schemas/util.sql: force_recreate(): recreate_users_cmds: use create_if_not_exists() rather than eval(), because cmd might manually re-create a deleted dependent view, causing it to already exist
Aaron Marcuse-Kubitza
05:27 PM Revision 13470: bugfix: schemas/util.sql: save_drop_view(): util.show_create_view(): don't include `OR REPLACE` because that causes nonuniform errors (eg. invalid_table_definition), instead of the standard duplicate_table exception caught by util.create_if_not_exists()
Aaron Marcuse-Kubitza
05:25 PM Revision 13469: fix: schemas/util.sql: show_create_view(): removed extra trailing ; because pg_get_viewdef() now includes one
Aaron Marcuse-Kubitza
05:24 PM Revision 13468: schemas/util.sql: show_create_view(): made inclusion of `OR REPLACE` configurable
Aaron Marcuse-Kubitza
05:10 PM Revision 13467: schemas/util.sql: save_drop_view(): use util.drop_view()
Aaron Marcuse-Kubitza
03:40 PM Revision 13466: /README.TXT: Maintenance: added steps to back up the vegbiendev databases
Aaron Marcuse-Kubitza
02:15 PM Revision 13465: bugfix: lib/runscripts/util.run: run_args_cmd(): don't prepend main to args if no args, because for a non-runscript, all args will be passed to main(), leading `main` to be doubled
Aaron Marcuse-Kubitza
01:30 PM Revision 13464: lib/tnrs.py: use the TNRS dev server (with private URL in tnrs.url) instead of the live server, since that contains datasources that we need
Aaron Marcuse-Kubitza
01:29 PM Revision 13463: lib/streams.py: added file_get_contents()
Aaron Marcuse-Kubitza
01:14 PM Revision 13462: lib/tnrs.py: configure the server separately from the base URL
Aaron Marcuse-Kubitza
01:12 PM Revision 13461: lib/: svn:ignore tnrs.url so the TNRS dev server URL does not become public
Aaron Marcuse-Kubitza

05/16/2014

12:03 AM Revision 13460: bugfix: web/.phpPgAdmin/.htaccess: only append subject if non-empty, to avoid confusing redirect.php
Aaron Marcuse-Kubitza

05/15/2014

11:57 PM Revision 13459: bugfix: web/.phpPgAdmin/.htaccess: custom pages: need to handle in query string because .php filename would be treated as path element
Aaron Marcuse-Kubitza
11:32 PM Revision 13458: web/.phpPgAdmin/.htaccess: support custom .php pages
Aaron Marcuse-Kubitza
10:30 AM Revision 13457: planning/meetings/BIEN conference call availability.xlsx: updated
Aaron Marcuse-Kubitza

05/13/2014

11:19 AM Revision 13456: schemas/util.sql: save_drop_views(): return text[] instead of concatenated text so that the queries will be run individually and be easier to debug
Aaron Marcuse-Kubitza
11:00 AM Revision 13455: schemas/util.sql: added eval(queries text[])
Aaron Marcuse-Kubitza
09:43 AM Revision 13454: bugfix: schemas/util.sql: force_recreate(): need to drop users in reverse to avoid dependencies that block the drop
Aaron Marcuse-Kubitza
09:25 AM Revision 13453: schemas/util.sql: added array_reverse()
Aaron Marcuse-Kubitza
09:25 AM Revision 13452: schemas/util.sql: added in_reverse()
Aaron Marcuse-Kubitza
08:39 AM Revision 13451: bugfix: schemas/util.sql: force_recreate(): need to use [[:lower:]], not [:lower:]
Aaron Marcuse-Kubitza
08:34 AM Revision 13450: schemas/util.sql: force_recreate(): debug-print PG_EXCEPTION_DETAIL, users to assist in debugging
Aaron Marcuse-Kubitza
08:33 AM Revision 13449: fix: schemas/util.sql: debug_print_value(): added trailing newline to visually distinguish the printed value
Aaron Marcuse-Kubitza
08:27 AM Revision 13448: schemas/util.sql: added debug_print_var()
Aaron Marcuse-Kubitza
07:06 AM Revision 13447: schemas/util.sql: debug_print_return_value(): use new util.debug_print_value()
Aaron Marcuse-Kubitza
06:59 AM Revision 13446: schemas/util.sql: added debug_print_value()
Aaron Marcuse-Kubitza
05:45 AM Revision 13445: fix: schemas/util.sql: util.force_recreate(): also support dropping things besides tables
Aaron Marcuse-Kubitza
04:50 AM Revision 13444: inputs/.TNRS/schema.sql: MatchedTaxon.Accepted_name_species: mapped to accepted_species_binomial
Aaron Marcuse-Kubitza
04:09 AM Revision 13443: fix: inputs/.TNRS/schema.sql: COMMENTs: always include newline before and after
Aaron Marcuse-Kubitza
03:50 AM Revision 13442: bugfix: schemas/public_.sql: _plots_20_tnrs_names: tnrs_accepted_*: use new accepted_*, *not* scrubbed_*, as scrubbed_* does *not* contain only the accepted name (as implied by the tnrs_accepted_* column name)
Aaron Marcuse-Kubitza
03:46 AM Revision 13441: bugfix: inputs/.TNRS/schema.sql: taxon_scrub, etc.: undid rename of accepted name columns to scrubbed_* (r13435), because these are actually *not* the same (scrubbed_* is the combination of accepted and no-opinion names). the accepted name columns will now be named accepted_*, following the standard naming scheme.
Aaron Marcuse-Kubitza
03:28 AM Revision 13440: bugfix: schemas/public_.sql: _plots_20_tnrs_names: morphospeciesSuffix must still be called morphospecies_suffix, not taxon_morphospecies, because that column has not been updated to the taxon_morphospecies yet (fixes r13435)
Aaron Marcuse-Kubitza
03:13 AM Revision 13439: fix: inputs/.TNRS/schema.sql: taxon_scrub, etc.: scrubbed_*: use columns from MatchedTaxon whenever possible, to as much as possible avoid the need to join to taxon_scrub.scrubbed_unique_taxon_name.*
Aaron Marcuse-Kubitza
02:53 AM Revision 13438: schemas/Makefile: vegbien.sql: exclude the same set of Source tables excluded by inputs/.TNRS/schema.sql, so that reinstalling TNRS doesn't change the contents of this file
Aaron Marcuse-Kubitza
02:29 AM Revision 13437: bugfix: inputs/.TNRS/grants.sql: added GRANT statements from schema.sql because these aren't run by `make inputs/.TNRS/reinstall`
Aaron Marcuse-Kubitza

05/12/2014

07:06 PM Revision 13436: lib/tnrs.py: retrieval_request_template: taxonomic_constraint, source_sorting: documented their meaning and why they need to be on/off
Aaron Marcuse-Kubitza
04:28 PM Revision 13435: schemas/public_.sql: _plots_20_tnrs_names: changed to set of columns requested by Martha
Aaron Marcuse-Kubitza

05/10/2014

09:32 PM Revision 13434: validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: updated to DB
Aaron Marcuse-Kubitza
09:17 PM Revision 13433: schemas/public_.sql: _plots_20_tnrs_names: added morphospecies suffix in order to also validate it
Aaron Marcuse-Kubitza
09:13 PM Revision 13432: schemas/public_.sql: _plots_20_tnrs_names: added verbatim name in order to also validate the formation of the concatenated name
Aaron Marcuse-Kubitza
08:49 PM Revision 13431: schemas/public_.sql: merged _plots_20_tnrs_input_names, _plots_21_tnrs_output_names into _plots_20_tnrs_names so the input and output names can be compared side-by-side
Aaron Marcuse-Kubitza
08:10 PM Revision 13430: schemas/vegbien.sql: synced to DB, which adds Source.errors table indexes added in the import
Aaron Marcuse-Kubitza

05/09/2014

07:27 PM Revision 13429: /README.TXT: Full database import: for test import, turn off DB backup: added () after after_import to show that it's a function
Aaron Marcuse-Kubitza
07:24 PM Revision 13428: /README.TXT: Full database import: refer to the shell instead of screen because it's also possible to use a nested shell for this
Aaron Marcuse-Kubitza
02:51 AM Revision 13427: bugfix: /README.TXT: Full database import: to import just a subset of the datasources: need to set @inputs separately from declaring it, because declare does not support () on Mac
Aaron Marcuse-Kubitza
02:16 AM Revision 13426: fix: /README.TXT: Full database import: on local machine: need to allow logging output to go to log files
Aaron Marcuse-Kubitza
02:05 AM Revision 13425: /README.TXT: Full database import: for test import, added step to turn off DB backup
Aaron Marcuse-Kubitza
01:48 AM Revision 13424: fix: /README.TXT: Full database import: setting $version: moved to inside screen to avoid affecting the outer shell
Aaron Marcuse-Kubitza
01:46 AM Revision 13423: /README.TXT: Full database import: screen: added alternative for a small import (just use nested shell)
Aaron Marcuse-Kubitza
01:42 AM Revision 13422: fix: /README.TXT: Full database import: clear any limit set in .profile: moved to inside screen because it must happen within screen to avoid affecting the outer shell
Aaron Marcuse-Kubitza
01:40 AM Revision 13421: fix: /README.TXT: Full database import: added step to clear any limit set in .profile (applicable to local machine)
Aaron Marcuse-Kubitza
01:35 AM Revision 13420: bugfix: schemas/Makefile: `%/install: public_.sql`: also need to replace `public` when preceded by ( , as it is in the validations queries
Aaron Marcuse-Kubitza

05/07/2014

08:14 PM Revision 13419: bugfix: /Makefile: install: use schemas/public/*re*install because schemas/public/install would errexit if the public schema already exists
Aaron Marcuse-Kubitza
07:17 PM Revision 13418: bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(datasrc), not $(schema), as $schema is only what this var is called in the runscripts
Aaron Marcuse-Kubitza
06:48 PM Revision 13417: bugfix: inputs/analytical_db/: need dummy table.run file to cause a schema to be created for this datasource
Aaron Marcuse-Kubitza
06:44 PM Revision 13416: fix: inputs/input.Makefile: $(sortFile): don't print the "add any missing tables to $(sortFile)" message every time the Makefile is run
Aaron Marcuse-Kubitza
06:44 PM Revision 13415: bugfix: inputs/input.Makefile: install: only run this for datasource dirs
Aaron Marcuse-Kubitza
05:18 PM Revision 13414: inputs/input.Makefile: install: use ./run's install target for clarity
Aaron Marcuse-Kubitza
05:18 PM Revision 13413: lib/runscripts/import.run: added install() target
Aaron Marcuse-Kubitza
04:56 PM Revision 13412: bugfix: inputs/input.Makefile: install: made it idempotent (using new $(datasrc_schema_exists)) so that it could be run by `make install` on an existing system
Aaron Marcuse-Kubitza
04:02 PM Revision 13411: bugfix: inputs/input.Makefile: $(datasrc_schema_exists): need to use $(shell ...)
Aaron Marcuse-Kubitza
03:31 PM Revision 13410: inputs/input.Makefile: added $(datasrc_schema_exists)
Aaron Marcuse-Kubitza
03:29 PM Revision 13409: lib/runscripts/in_datasrc_dir.run: use new local.run
Aaron Marcuse-Kubitza
03:25 PM Revision 13408: added lib/runscripts/local.run
Aaron Marcuse-Kubitza

05/06/2014

03:39 PM Revision 13407: web/links/index.htm: updated to Firefox bookmarks: iPlant links: fixed broken links
Aaron Marcuse-Kubitza
03:34 PM Revision 13406: web/links/index.htm: updated to Firefox bookmarks: starscream links: removed favicons so they wouldn't wake the Mac up on network access whenever someone opens the links page
Aaron Marcuse-Kubitza
 

Also available in: Atom