import_all: Run the import directly into a new, already-versioned public schema. This removes the need to manually rename the schema after import, and allows the backup commands to use the stored $version shell variable to refer to the last import.

schemas/Makefile: %/publish: Added instruction to run `unset version` after the command, to clear the $version shell variable which will be set by import_all

README.TXT: Data import: Replaced <import_name> with <version> because the import name is now just the version

README.TXT: Data import: Replaced r<revision> with <version> because the version string is now equal to r<revision>

README.TXT: Backups: Replaced <date> with <version> because the date is no longer included in the version string

README.TXT: Name archived imports without the "public." prefix so that their backups will work with the new `make backups/%.backup/remove` command, which does not add back the prefix

backups/Makefile; $(public*): Don't add a "public." prefix to get the name of the public schema

backups/Makefile: Removed no longer used $(rmSchema)

backups/Makefile: Use \$(rmSchemaCmd) from lib/common.Makefile instead of \$(rmSchema)

vegbien_dest: Use $version as $public when $public not provided. When neither is provided, continue to use "public" and also set $version to that.

schemas/Makefile: Installation: rotate: Use just the version, without the "public." prefix

schemas/Makefile: Installation: `public/install public%/install`: Generalized to /install to allow public schema versions with any name. This requires moving `/install: %.sql` before it to override it.

schemas/Makefile: Installation: Merged public/install and public%/install

schemas/Makefile: Installation: Moved %/uninstall to beginning of section because it applies to all schemas

schemas/Makefile: Installation: public: Generalized public%/publish to %/publish so that public schema versions don't have to start with public_

schemas/Makefile: Installation: %/uninstall: Also display schema delete confirmation for schemas whose name is just the version suffix (r<revision #>)

schemas/Makefile: Merged public%/uninstall and %/uninstall

lib/common.Makefile: Added version target, which prints the current $(version) value

schemas/Makefile: Installation: public: public%/uninstall: Fixed bug where need to remove the specified version of the public schema, not public itself. Generalized $(confirmRmPublicSchema) so it could also be used for named versions of the public schema. Inlined $(rmPublicSchema) since it's now only used in one place.

lib/common.Makefile: Revisions: $(version): Use just the revision # to avoid cluttering the schema and log file names with long datetime strings

schemas/Makefile: public%/install: schema comment: Include current date/time after version

lib/common.Makefile: Replaced no longer used $(date) with function to generate human-readable text date (rather than date to put in filename). Removed leading zeros from date and hour. Added timezone.

backups/Makefile: Removed no longer used $(dateFmt), $(mtime)

backups/Makefile: Removed %.backup/rotate, because this incorrectly causes the current time rather than the version to be used in the backup filename. The version should instead be specified in the backup filename when it's created.

schemas/Makefile: Installation: public: Added public%/publish to replace the current public schema with the given version

schemas/Makefile: Installation: public: public/uninstall: Added public%/uninstall as a target to allow uninstalling versions of the public schema

schemas/Makefile: Installation: public: public%/install: Add a comment on the schema containing the versioned schema name, so that if the schema is later renamed to just public (i.e. "published" as the current version), it will still be possible to tell which version the public schema came from

schemas/Makefile: Installation: public: Added public%/install, to install a version of the public schema

schemas/Makefile: Removed unused $(os)

schemas/Makefile: Removed unused $(SED)

Moved schemas-related commands from root Makefile to schemas/Makefile

Makefiles: Factored out common vars/functions into lib/common.Makefile

root Makefile: $(psqlNoSearchPath): Merged $(psqlAsBien) into it because it's the only place $(psqlAsBien) is used

root Makefile: $(psqlAsBien): Use psql_script_vegbien instead of psql_vegbien, which adds $(psqlOpts) itself

schemas/Makefile: Include lib/common.Makefile

inputs/import.stats.xls: Reformatted so the first by column import and the comparison by row import will fit on the same page when printed on portrait-mode letter paper

inputs/import.stats.xls: Changed import type labels to By row/By column so they would fit into one field, leaving the extra field free to contain the revision #

lib/common.Makefile: Revisions: Allow $(version) to be overridden in the environment, so that the public schema and all log files share the same, pregenerated version

schemas/vegbien.sql: Merged provider_view, provider_count, and owner_count into provider_count, using the combining query for Brad's data providers page at <http://bien.nceas.ucsb.edu/bien/people/data-providers/>

schemas/vegbien.sql: sync_taxon_trait_to_view(): Changed pkey to index because there can be multiple values of the same taxon's trait from different observations

mappings/Makefile: VegCore.csv: Filter out the VegCore tables so they are not matched as terms. This is necessary because some terms have the same name as a table, but the term should be the match rather than the table.

sql.py: DbConn.col_info(): raising sql_gen.NoUnderlyingTableException: Fixed bug where also need to catch DoesNotExistException, which is thrown by ::regclass

sql.py: DbConn.col_info(): Fixed bug where need to run run_query() recoverably, because this query throws an exception if the column's table does not exist (the information_schema query just returned no rows)

sql.py: DbConn.col_info(): Fixed bug where need to use pg_get_expr() on pg_attrdef.adbin instead of shortcut field adsrc, because adsrc does not include schema qualifiers on table names (including strings passed to `nextval('..._seq'::regclass)`)

sql.py: DbConn.col_info(): Fixed bug where need to pass through cacheable param to run_query()

sql.py: DbConn.col_info(): Fixed bug where need to use .to_str(self) instead of self.esc_value() because self.esc_value() expects a value, not a sql_gen.Literal instance

sql.py: DbConn.col_info(): Fixed bug where self needs to be used everywhere that db normally is, because this is a DbConn method rather than a global function

sql.py: DbConn.col_info(): For PostgreSQL, use pg_catalog tables directly instead of their views in information_schema. This allows using ::regclass to look up the table in the search_path, and fixes a bug in imports with an explicit public schema where column types were looked up in public instead of public.<version>. Also don't wrap default using sql_gen.as_Code() when it's None (indicating no default value, aka default=NULL), because this value is interpreted specially by sql_gen.TypedCol.

inputs/Makefile: Input data: $(rsyncSrcs): Also include log files other than install.log.sql

import_all: Run all imports (not just the main datasources' import) with $import_source turned off, so that the Source tables will not be imported a second time when the datasource's main tables are imported. Note that it's not necessary to wait for asynchronous commands after the jobs for the main import are started (so that $import_source is not unset until after they are started), because with_all does not return until all jobs are started and have noted the $import_source setting in effect in the shell environment.

import_all: Source tables import: Fixed bug where need to use $all option to with_all to also include special datasources starting with "."

make_analytical_db: Also create taxon_trait materialized view

inputs/*/*/map.csv: Reverted special OMIT mappings for input columns that have the same name as a VegCore table and have not yet been mapped to a VegCore term

mappings/Makefile: VegCore.csv: Filter out the VegCore tables so they are not matched as terms. This is necessary because some terms have the same name as a table, but the term should be the match rather than the table.

mappings/VegCore.csv: Changed line endings to \r\n to match the output of filter_out_ci

inputs/CTFS/TaxonOccurrence/map.csv: Mapped SpeciesAuthority

backups/Makefile: Synchronization: $(remote): Fixed bug where need trailing / at end of path

backups/Makefile: Synchronization: $(remote): Updated path to backups

README.TXT: Data import: On jupiter: Updated path to backups

README.TXT: Installation: Added command to change to the directory of the checked out files

README.TXT: Installation: Added command to check out files from svn

schemas/vegbien.sql: Added taxon_trait materialized view

mappings/Veg+-VegCore.csv: Sources: Removed redundant bien2_ prefix from bien2_staging subnamespace

schemas/vegbien.sql: trait: trait_unique: Removed value and units because there should only be one value of a trait for each taxonoccurrence

schemas/vegbien.sql: Reattached trait to taxonoccurrence instead of taxonlabel, because the TraitObservation traits data is actually associated with a particular occurrence (plant observation complete with location, date, etc.), rather than just a taxon

Added inputs/bien2_traits/

mappings/VegCore-VegBIEN.csv: Mapped traits-related DwC terms measurementType, measurementValue, measurementUnit

schemas/vegbien.ERD.mwb: Added trait table to ERD

schemas/vegbien.sql: trait: Added trait_unique unique index

schemas/vegbien.sql: trait: Added units field

schemas/vegbien.sql: trait: Renamed type to name because TraitObservation stores trait names rather than types

schemas/vegbien.sql: trait: Linked to taxonlabel instead of stemobservation, because TraitObservation's traits are taxon-level and stem-level traits currently go in named fields instead of a stem traits table

inputs/.TNRS/tnrs_*/map.csv: Remapped Source to OMIT so it won't match to the Source table

inputs/.TNRS/tnrs_other/map.csv: Updated for new VegCore terms, which include Source as a table name. This field will need to be remapped so it doesn't collide with the table name.

inputs/import.stats.xls: Updated import times

README.TXT: Data import: Added step to check that the source table contains entries for all inputs

Regenerated vegbien.ERD exports

make_analytical_db: Also populate owner_count

make_analytical_db: Generate provider_count before analytical_aggregate because it's much faster

schemas/vegbien.sql: Added materialized view owner_count, generated from owner_count_view

make_analytical_db: Also populate provider_count

schemas/vegbien.sql: Added materialized view provider_count, generated from provider_count_view

schemas/vegbien.sql: Added provider_count_view for counts of occurrences per top-level provider

Regenerated mappings/VegCore.htm

Regenerated mappings/VegCore.htm

Regenerated mappings/VegCore.htm

schemas/vegbien.sql: provider_view: Sort NULL sourcetype last

schemas/vegbien.sql: Added provider_view, which combines source and sourcename

schemas/vegbien.sql: sourcename: Gave public_ SELECT permissions

Regenerated mappings/VegCore.htm

README.TXT: Maintenance: VegCore data dictionary: Regenerate everything in mappings/ that changes when VegCore.htm changes (such as VegCore.tables.redmine) instead of just VegCore.csv

inputs/*/Source/map.csv without mappings: Added referenceType, etc. mappings. This also ensures that the source table entry for the datasource will be created before the herbaria list is imported, causing all top-level datasources to sort at the top of the source table.

schemas/vegbien.sql: Granted the public_ user read-only access to the contents of the source table

root Makefile: PostgreSQL: $(editPhppgadmin): Ignore errors if patch has already been applied

lib/phpPgAdmin.config.inc.php.diff: Remove context so segment matching would depend only on the $conf['extra_login_security'] line itself

mappings/Makefile: Added VegCore.tables.redmine, which contains the Redmine-formatted list of VegCore tables to paste into <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Tables>

mappings/: Removed no longer used VegCore.redmine. VegCore.csv is now generated from the Redmine page instead of the other way around.

mappings/Makefile: Added VegCore.tables.csv, which contains all the tables in the VegCore data dictionary

README.TXT: Data import: backups/fix_perms: Run using sudo to also change permissions on files owned by the bien user, and to change the owner of files owned by you to the bien user

Regenerated mappings/VegCore.csv, which adds categories