lib/sql_io.py: automatic handling of input/output column type mismatches: also do this for identifying columns, which first cause an error in a join in sql.distinct_table() rather than in the main insert (and thus were not handled by the existing error handling). previously, the user would have had to manually cast the input column in postprocess.sql. this involves getting handle_MissingCastException() to update join_cols as well as mapping.
lib/sql.py: distinct_table(): don't cache the table creation, because this prevents different distinct_tables from being created for the same input table
lib/sql.py: create_table(), copy_table_struct(): support custom query options, such as cacheable
lib/sql_io.py: put_table(): main loop MissingCastException handler: factored out into nested function so that it can also be used elsewhere
lib/sql.py: parse_exception(): parse "operator does not exist" errors as MissingCastExceptions (these appear when a staging table column is of the wrong type)
bugfix: /README.TXT: Full database import: Check that source contains [# datasources] rows up through XAL: added alternative verification method when this is not the case (some datasources may be near the end depending on import order)
schemas/vegbien.sql: public_validation: split into separate subschemas for plots, specimens, traits
schemas/Makefile: vegbien.sql: support auxiliary schemas besides *_validation
fix: schemas/Makefile: %/publish: removed no longer applicable instructions about running `make schemas/rotate`. you should not publish a schema until you are satisfied that it can replace the previous public schema, so that the previous public schema doesn't need to be saved.
fix: schemas/vegbien.sql: public_validation comment: undid incorrect replacement of "validation" with "public_validation"
bugfix: schemas/Makefile: "public": dont
schemas/Makefile: %/publish: don't ignore errors if schema doesn't exist, because it should always exist if you are publishing it
schemas/Makefile: `%/uninstall: vegbien.sql`: use util.schema_bundle_rm() instead of public.rm(), because when the public schema is incompletely imported, public.rm() will not yet exist
schemas/Makefile: `%/install: vegbien.sql`: support auxiliary schemas besides *_validation
schemas/Makefile: %/uninstall, %/publish: use new public.rm(), publish(), which encapsulate the process of renaming multiple related schemas and also allow additional auxiliary schemas beyond just *_validation
schemas/vegbien.sql: added rm(), which deletes this schema
schemas/vegbien.sql: added publish(), which publishes this schema
schemas/vegbien.sql: added public_schema_publish()
schemas/util.sql: added functions for maintaining schema bundles, groups of schemas with a common prefix
schemas/util.sql: added schema_bundle_get_schemas()
schemas/util.sql: added schema_replace()
schemas/util.sql: added schema_rm(), schema_rename()
schemas/vegbien.sql: datasource_publish(): removed nested transaction, which is not needed because Postgres runs each top-level command in a transaction (including in autocommit mode)
bugfix: schemas/Makefile,vegbien.sql: renamed validation to public_validation to match the suffixed name applied by `make schemas/public/install`
schemas/Makefile: %/install, %/publish for public schema: added usage
/README.TXT: to back up the version history: added back `git svn fetch` so we keep the git export up-to-date, too
/README.TXT: to back up the version history: added runtimes (1.5 h for the initial svnsync)
/README.TXT: to back up the version history: added trailing /s to dirs
bugfix: /README.TXT: to back up the version history: fixed svn_repo/ path
**/LICENSE.TXT: corrected "it recommended" to "it is recommended"
_license/non-open-source/applies_to.txt: deleted items: tip revision needs to refer to the revision at which the problem was discovered, not the revision before that
_license/non-open-source/applies_to.txt: deleted items: clarified that some things deleted after r11705 (when we started copyright scrubbing) might also be non-open-source, because they might have been deleted for reasons other than copyright scrubbing and thus would be missing the "copyright scrub" tag
/LICENSE.TXT: removed separate non-open-source disclaimer about parts of the version history before r11705, because this is now listed in _license/non-open-source/applies_to.txt
_license/non-open-source/applies_to.txt: included deleted items that are non-open-source, so that the list correctly applies to the entire repository (which includes the version history)
**/LICENSE.TXT: disclaimer about things that are not open-source: removed "these are copyright their respective owners", because this is understood
_license/README.TXT: changed "parts" to "components" for clarity
_license/non-open-source/applies_to.txt: UArizona?: added BIEN2/traits_observation_counts.xls
/README.TXT: to back up the version history: use svnsync instead of `git svn fetch`, so that the backup is in a format that can be directly reimported into an svn repo
/LICENSE.TXT: parts of the repository that are not open-source: added ,s to clarify the meaning
/LICENSE.TXT: documented that this license refers specifically to the repository hosted at https://code.nceas.ucsb.edu/code/projects/bien/trunk/, since we also have a publicly-accessible checkout at http://fs.vegpath.org/, which contains numerous non-open-source files (more than are listed in _license/non-open-source/applies_to.txt)
fix: inputs/VegBIEN/Redmine/wiki/.htaccess: redirect to new main page when accessed without trailing /
web/BIEN3: redirected to .wiki since the wiki homepage now contains the list of all BIEN3 components
web/links/index.htm: updated to Firefox bookmarks. Quick links: use bien3.vegpath.org to link to the main BIEN3 page
added web/BIEN3 symlink, with .bien3 case-insensitive alias
_license/*/applies_to.txt: added exclusion for "UCSB modifications"
added _license/GPL2.txt.url
/LICENSE.TXT, _license/VegBank_schema/LICENSE.TXT: moved text of the GPL2 license into separate file _license/GPL2.txt to avoid cluttering up LICENSE.TXT
_license/*/LICENSE.TXT: documented that the first section of the file is from license.eml.txt in each folder
/LICENSE.TXT: GPL text: include the full range of copyright years
/LICENSE.TXT: added disclaimer that parts of the version history before r11705 (2013-11-21) are NOT open-source. this is the revision of the first copyright scrub.
added issues/ with LICENSE.TXT for the issue tracker (BSD)
wiki/_license/*/: symlink the contents instead of the folders, because the root dir's and wiki's applies_to.txt's will be different
wiki/LICENSE.TXT: documented what this license refers to (the wiki hosted at https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/)
wiki/LICENSE.TXT: added message about including _license/ when redistributing, as for /LICENSE.TXT
wiki/_license/LICENSE.TXT: replaced with README.TXT as in /_license
wiki/LICENSE.TXT: changed "the page text of the wiki" to "the wiki's components" since some of the attachments will eventually be marked as open-source
/LICENSE.TXT: moved emphasized sections to top
/LICENSE.TXT: changed "the open-source parts of the repository" to "the repository's components" so that this sentence makes sense once the non-open-source exception is removed
/LICENSE.TXT: documented that the non-open-source files are now listed in _license/non-open-source/applies_to.txt
added _license/non-open-source/applies_to.txt
removed schemas/VegX/run.py, which is a duplicate of _archive/bienDjango/trunk/bien/v/models.py (this may have ended up here because it was a wiki page attachment, but the initial revision did not indicate the source)
added _license/VegBank_schema/applies_to.txt
schemas/VegCore/vegbien_identifier_examples.xlsx: changed to symlink to Brad_Boyle/vegbien_identifier_examples.xlsx, to avoid duplicate copies
added _license/TWiki/ (for inputs/VegBIEN/TWiki/)
added _license/PostgreSQL_config (for schemas/*.conf)
*/LICENSE.TXT: GPL: removed divider between the copyright header and license text, since these are part of the same license
_license/ESA/: renamed to VegBank_schema/ because this license applies only to the VegBank schema
_license/README.TXT: merged ¶2 into ¶1
_license/README.TXT: replaced "authority information proving the open-source status" with the simpler "open-source proof". clarified that each LICENSE.TXT is meant as a proof.
/LICENSE.TXT: added note that when redistributing, it recommended to include _license/, so that the open-source proof is included along with the license text (this information should be here as well as in _license/README.TXT because redistributors might not think to look in that file)
/LICENSE.TXT: removed "svn" before "repository" because this would apply equally to a git export of the repository
/LICENSE.TXT: replaced "portions" with "parts" for consistency with the _license/README.TXT wording
_license/README.TXT: clarified that this folder contains authority information proving the open-source status of parts of the codebase. added recommendation to include this folder when redistributing, so that the open-source proof is included along with the license text.
_license/LICENSE.TXT: renamed to README.TXT because this actually contains a description of the folder, not license text
schemas/Makefile: removed no longer used rename/%, rotate. use %/install, %/publish instead to manage public schema versions.
schemas/Makefile: removed no longer needed public/uninstall, which is handled more generally by `%/uninstall: vegbien.sql` instead
schemas/Makefile: `%/uninstall: vegbien.sql`: added support for *_validation schema
schemas/Makefile: %/uninstall: split into 2 rules like for %/install, so that different actions can easily be added for the public schema
schemas/Makefile: %/publish: support auxiliary *_validation schemas
bugfix: schemas/Makefile: `%/install: vegbien.sql`: need -e before each s/// expression
schemas/Makefile: `%/install: vegbien.sql`: create a custom _validation schema for *each public schema, rather than one for the most recently-created public schema. this allows validations to continue to be run against a previous version of the DB while a new version is being imported.
validation/aggregating/traits/bien3_validations_traits_bien3.sql: include only records from the bien2_traits datasource, to ensure that the scope of all queries is limited to the datasource
validation/aggregating/*/*.sql (the output queries): added `SET search_path TO validation`
validation/aggregating/specimens/qualitative_validations_specimens.sql: added CREATE VIEW prefixes using the steps at wiki.vegpath.org/Aggregating_validations_refactoring#prepend-CREATE-VIEW
validation/aggregating/traits/bien3_validations_traits_bien3.sql: added CREATE VIEW prefixes using the steps at wiki.vegpath.org/Aggregating_validations_refactoring#prepend-CREATE-VIEW
validation/aggregating/plots/bien3_validations_salvias_vegbien.sql: added CREATE VIEW prefixes using the steps at wiki.vegpath.org/Aggregating_validations_refactoring#prepend-CREATE-VIEW
added validation/aggregating/specimens/qualitative_validations_specimens.sql template, with query descriptions from NY/qualitative_validations_source_db_NYBG.sql
validation/**: removed old versions of by=-tagged files. these were not renamed in the last commit because there were newer versions that they collided with. note that because , sorts before . , the newer version (with more ",name@inst" tags) was correctly renamed instead of the older version.
validation/**: removed by=... indicators so that we don't have to add an author name every time a file is changed. the author indicator is not needed now that both Brad's and my edits are open-source.
inputs/bien2_traits/TraitObservation/postprocess.sql: remove rows with no taxon name, which are invalid, and which helps simplify the aggregating validations queries
added validation/aggregating/pipeline/aggregating_validations_pipeline.odg with .png export
added validation/aggregating/plots/FIA/bien3_validations_fia_input.sql.eml.txt. note that a .url file is not needed because this information is in the .eml.txt file.
fix: validation/aggregating/traits/_archive/bien3_validations_traits_bien3.by=Brad@iPlant.sql: renamed to indicate that this is actually not Brad's original file, as the filename would seem to indicate
added validation/aggregating/plots/FIA/
validation/aggregating/*.by=Brad@iPlant*: placed under version control, since these are now open-source
validation/**: removed svn:ignore exclusion of iPlant-owned files, since these are now open-source
