inputs/SALVIAS/_MySQL/: Updated svn:ignore from running `make inputs/SALVIAS/add`
mappings/VegCore-VegBIEN.csv: matched place's coordinates: Fixed bug where coordinates entry itself needed to have its datasource (reference) set to geoscrub, in addition to the place entry that uses it, in order to match up properly with geoscrub's corresponding input place (whose coordinates as well as place are owned by the geoscrub datasource)
mappings/VegCore-VegBIEN.csv: matched place's coordinates: Fixed bug where coordinates mappings with and without matched_place_id=0 need to sort together in order to be merged, by prepending ".," to the place attrs list
inputs/VegBank/plot_/test.xml.ref: Updated inserted row count
inputs/import.stats.xls: Updated import times
Regenerated vegbien.ERD exports
inputs/Makefile: Input data: $(rsyncLogs): Also include logs from the datasource's top-level logs/ dir, which contains make_analytical_db.log.sql
inputs/VegBank/vegbank.~.clean_up.sql: Remove still-embargoed plots
inputs/VegBank/vegbank.~.clean_up.sql: Enable cascading deletes by setting all foreign keys to ON DELETE CASCADE
Added inputs/VegBank/_src/vegbank.schema.sql.make and vegbank.schema.sql
input.Makefile: Staging tables installation: sql/install: Use new pg_dump_limit to remove security and schema-setting commands
Added pg_dump_limit to filter a PostgreSQL DB dump to remove security and schema-setting commands
inputs/.geoscrub/geoscrub_cleaned_unique/create.sql: Removed no longer needed index on latitudeDecimalVerbatim, longitudeDecimalVerbatim, which is now on geoscrub_cultivated instead
schemas/vegbien.sql: analytical_stem_view: Fixed bug where needed to join on new geoscrub_cultivated, not geoscrub, for all geoscrub-related information. geoscrub contains many duplicate records, causing one input row to match many rows in geoscrub, when there should only be one entry for each coordinate pair.
Added inputs/.geoscrub/geoscrub_cultivated/
inputs/.geoscrub/geoscrub_cleaned_unique/create.sql: Added index on latitudeDecimalVerbatim, longitudeDecimalVerbatim for use by analytical_stem_view
inputs/newWorld/geoscrub.schema.~.changes.sql: Change countryNameStd type to text to allow merge-joining with place.country in analytical_stem_view
inputs/newWorld/geoscrub.schema.~.changes.sql: ALTER TABLE ... ALTER COLUMN statement: Reformatted to allow adding additional ALTER COLUMN clauses
inputs/.geoscrub/geoscrub_cleaned_unique/create.sql: Change latitudeDecimalVerbatim, longitudeDecimalVerbatim types to double precision to allow merge-joining with coordinates.latitude_deg, longitude_deg in analytical_stem_view
README.TXT: Data import: Instead of using `make schemas/rotate` and then renaming the public schema to the correct name, just rename directly to the correct name using `make schemas/rename/...`. Use new import_name to determine the import name instead of manually finding the date in the first datasource's log file name.
Added import_name, which gets the name of an import based on its log file names
README.TXT: Data import: Moved checking that imports were successful before running make_analytical_db
root Makefile: Installation: Fixed bug where schemas/install needed to happen after inputs/install because some of the public schema's views now depend on inputs
schemas/vegbien.sql: analytical_stem_view: cultivatedBasis: Concatenate ''::text to geoscrub.isCultivatedReason so it will be cast to a text field both on PostgreSQL 9.1.1 (local machine), which removes any explicit cast to text when creating the view, and 9.1.6 (vegbiendev), which requires an explicit cast to text
schemas/vegbien.sql: analytical_stem_view: cultivatedBasis: Use geoscrub.isCultivatedReason instead when geoscrub.isCultivated is used as the source for cultivated
schemas/vegbien.sql: analytical_stem_view: Use geoscrub.isCultivated when taxonoccurrence.iscultivated is not provided (joining to geoscrub on the coordinates)
root Makefile: VegBIEN DB: Schemas: Run all schema installs and uninstalls using no_search_path=1, so that the schemas in the automatic search_path are not required for the command to run
psql_vegbien: Added $no_search_path option to turn off the automatic SET search_path directive
schemas/vegbien.sql: taxonverbatim: Added growthform for growthform based on the taxon name rather than provided with the input data's taxonoccurrence
schemas/vegbien.ERD.mwb: Fixed lines
inputs/SALVIAS/plotMetadata/: LEFT JOINed with lookup_MethodCode to create plotMetadata_
schemas/vegbien.sql: threatened_taxonlabel_view: Fixed bug where needed DISTINCT on resulting taxonlabel_id because some descendants apparently appear in multiple threatened taxonlabels' subtrees
schemas/vegbien.sql: analytical_*: Added threatened, using new threatened_taxonlabel lookup table
schemas/vegbien.sql: reference_by_shortname(): Fixed bug where need to use $-syntax to reference params in sql-language functions
schemas/vegbien.sql: threatened_taxonlabel_view: Use new reference_by_shortname()
root Makefile: VegBIEN DB: Schemas: public: schemas/public/uninstall: Fixed bug where need to run psql_vegbien without public in the search_path because it may have already been deleted
root Makefile: VegBIEN DB: Schemas: public: schemas/public/install: Fixed bug where need to run psql_vegbien without public in the search_path because it doesn't exist, by setting public to the empty string (deleting it)
vegbien_dest: $schemas: Don't include the , before $public if it has been set to the empty string (deleted)
schemas/vegbien.sql: Added reference_by_shortname(). Using this function instead of the manual query should force the query planner to evaluate this expression first, rather than complexly reordering joins to place this nested select as a filter condition.
schemas/vegbien.sql: Added threatened_taxonlabel derived table with generating view threatened_taxonlabel_view
Updated inputs/UNCC/Specimen/test.xml.ref inserted row count
mappings/VegCore.csv: Added threatened
inputs/VegBank/vegbank.~.clean_up.sql: Remove private columns (plot.reallatitude, reallongitude) that should not be publicly visible
inputs/CVS/Organism/map.csv: Removed now-dropped realLatitude, realLongitude
Added inputs/CVS/Organism/postprocess.sql to drop private realLatitude, realLongitude columns
input.Makefile: Staging tables installation: Added back postprocess.sql, which is now used for one-time dropping of private columns that should not be publicly visible
input.Makefile: Maps building: %/.map.csv.last_cleanup: $(dict) canon/translate: Use new $(translate?)
input.Makefile: Maps building: %/.map.csv.last_cleanup: Added $(srcDict) as a prerequisite, so that .last_cleanup will be re-run if it changes. Added empty $(srcDict) target in case it doesn't exist.
inputs/bien_web/observation/map.csv: Omit *_index because they are placeholder columns created by the MySQL to PostgreSQL translation
input.Makefile: Maps building: %/.map.csv.last_cleanup: Fixed bug where can only canon/translate using $(srcDict) if it exists for the datasource
inputs/bien_web/observation/: Regenerated from actual bien_web.observation schema on nimoy, which has additional columns
input.Makefile: SVN: $(svnFilesGlob): Added top-level map.csv, which can be used to apply a datasource-global data dictionary to all tables
input.Makefile: Maps building: %/.map.csv.last_cleanup: Also apply any map.csv at the top level of the datasource directory. This can be used to apply a datasource-global data dictionary to all tables.
my2pg: Also remove column comments. Note that these cannot be translated by sed, because PostgreSQL only allows setting column comments in a separate statement, not inline with the column's entry in the CREATE TABLE statement, and sed can only make replacements contiguous with the input line.
mappings/VegCore.csv: Removed incorrect duplicate entry for verbatimSubgenus
schemas/vegbien.sql: _taxon_family_require_std(): Fixed bug where name needed to be lowercased before checking if it ended in -aceae, to support family names that are uppercase. Note that this resulted in the family not being prepended to the TNRS input name for datasources with uppercase family names, so the next DB import will likely produce a number of unscrubbed TNRS input names which now have the uppercase family prepended.
inputs/.TNRS/tnrs/tnrs.make: Fixed bug where need to reference the log file path relative to the make script itself, because otherwise the log file would go in inputs/.TNRS/logs/tnrs.make.log.sql
inputs/.TNRS/tnrs/tnrs.make: Fixed bug where need to use just the basename of $0 as the log file name
Added inputs/IUCN/
input.Makefile: SVN: add: Added _src/ (when it exists). $(_svnFilesGlob): Added .url, .pdf files in _src/.
psql_vegbien: Use bash because it supports substitutions
psql_vegbien: Set the search_path to $out_schemas set by vegbien_dest. This will enable running any psql_vegbien script on a schema other than public.
schemas/vegbien.sql: analytical_stem_view: Changed inner joins on non-datasource taxonlabels to LEFT JOINs, to ensure that an entry is included in the analytical DB even if there was no matched taxonlabel. In theory, this shouldn't be necessary, because every taxonlabel should have a canonical taxonlabel since canon_label_id is auto-populated to the taxonlabel_id if there is no matched_label_id; there should be a taxonverbatim for every datasource and accepted taxonlabel because datasources link to taxonlabel via taxonverbatim and TNRS populates a taxonverbatim for every accepted taxonlabel; and there should be a parsed taxonlabel for every datasource taxonlabel because the mappings populate it.
schemas/vegbien.sql: analytical_stem_view: Removed join on specimenreplicate, because it isn't used in the analytical DB. Each specimen will still get an entry in analytical_*, because it gets its own location.
README.TXT: Data import: Before starting import, added step to run `make inputs/upload` and reinstall newly-uploaded datasources
README.TXT: Schema changes: Remember to update the following files with any renamings: Added mappings/verify.*.sql
README.TXT: Data import: make_analytical_db: Documented how to view progress in log file
make_analytical_db: Run all commands synchronously so the log file output doesn't become jumbled
make_analytical_db: Fixed bug where log file needed to be appended to instead of overwritten
make_analytical_db: Wrap each individual command in `set -x` to avoid echoing low-level commands such as sleep, wait
make_analytical_db: Moved log file to inputs/analytical_db/logs/make_analytical_db.log.sql so it would be synced along with the other import logs
inputs/.TNRS/tnrs/tnrs.make: Output the time at which it's run, so this is included in the log file
inputs/.TNRS/tnrs/tnrs.make: Moved log file to logs/tnrs.make.log.sql so it would automatically be synced along with the other import logs
inputs/Makefile: Import logs: $(rsyncLogs): Always download the TNRS daemon log, rather than requiring tnrs_log=1 to be specified to download it
make_analytical_db: Output the time at which it's run, so this is included in the log file
make_analytical_db: Store output in schemas/make_analytical_db.log
schemas/vegbien.sql: Removed no longer used make_analytical_db(). Use bin/make_analytical_db instead.
make_analytical_db: Use new psql_verbose_vegbien
Added psql_verbose_vegbien
make_analytical_db: Use psql_script_vegbien, which contains the necessary psq options, instead of setting those options manually
make_analytical_db: Run the SQL commands directly with psql instead of using the make_analytical_db() function. This provides incremental results and avoids running all commands in one transaction, thus preventing pgAdmin from freezing when the user attempts to access a table used in analytical DB creation (because the TRUNCATE statement fully locks the table until the entire analytical DB is built).
schemas/vegbien.sql: make_analytical_db(): Added make_family_higher_plant_group()
inputs/import.stats.xls: Updated import times. Fixed input row counts and import times to include derived data, such as TNRS and geoscrub, which adds to the import time and therefore should be considered in the import's speed. (TNRS was already being included in the import time for some, but not all, imports.)
schemas/vegbien.sql: analytical_*: Added higherPlantGroup
pg_dump_vegbien: When not in data mode, use --inserts so the INSERT statements generated can be run directly in pgAdmin, they can be directly translated to MySQL, and an empty COPY FROM statement isn't generated for every table
schemas/vegbien.sql: Added family_higher_plant_group lookup table, generated with make_family_higher_plant_group() using the contents of higher_plant_group_nodes
mappings/VegCore-VegBIEN.csv: Always map taxonNameOrEpithet to taxonomicname, now that it's globally unique at all ranks in the datasource that provides it (NCBI)
inputs/.NCBI/nodes/create.sql: Make name_txt completely globally unique by removing all duplicates, not just duplicate genera
inputs/.NCBI/nodes/create.sql: Make name_txt (mostly) globally unique by removing several other kingdoms/superkingdoms, not just Animalia
inputs/.NCBI/nodes/create.sql: Making genus globally unique: Moved comment with kingdom name to line with DELETE, and put "delete cascades to descendants" comment on its own line
schemas/vegbien.sql: Added higher_plant_group_nodes lookup table
schemas/Makefile: DDLs: $(pg_dump): Turn off schema-only mode so that lookup table contents are included as well
pg_dump_vegbien: Except in (raw) data mode, filter out pg_catalog.setval() statements (only lookup table contents should be preserved)
lib/PostgreSQL-MySQL.csv: Remove SELECT pg_catalog.setval() statements
lib/PostgreSQL-MySQL.csv: Remove COPY statements
pg_dump_vegbien: $schema flag defaults to on to export only schema
