Activity
From 12/13/2012 to 01/11/2013
01/11/2013
- 05:05 PM Revision 7205: README.TXT: Data import: Deleting imports before the last: Added instructions to keep a previous import instead of deleting it
- 04:22 PM Revision 7204: input.Makefile: Staging tables installation: $(logInstall): Always log the installation, regardless of the $log env var, because $log is set by default on development machines but an install log should still be created
- 01:03 PM Revision 7203: schemas/vegbien.ERD.mwb: Regenerated exports
- 10:19 AM Revision 7202: schemas/vegbien.sql: unscrubbed_taxondetermination_view: Fixed bug where need to handle the case where (SELECT source.source_id FROM source WHERE source.shortname = 'TNRS') is NULL because no TNRS names have been imported yet
- 09:44 AM Revision 7201: **/new_terms.csv, **/unmapped_terms.csv: Regenerated using `make missing_mappings`
- 09:19 AM Revision 7200: mappings/VegCore-VegBIEN.csv: morphoname: Remapped to the original rather than current taxondetermination because this is the *original* name applied by the author
- 09:16 AM Revision 7199: inputs/SALVIAS*/Organism/map.csv: Remapped voucher_string/coll_number to recordNumber instead of catalogNumber, because this number is actually applied by the collector rather than by a herbarium
- 09:11 AM Revision 7198: mappings/VegCore-VegBIEN.csv: Mapped recordNumber to new specimenreplicate.collectionnumber
- 09:02 AM Revision 7197: mappings/VegCore-VegBIEN.csv: Also map recordNumber (collectionnumber) to the indirect voucher's specimenreplicate
- 08:48 AM Revision 7196: inputs/*/*/map.csv: Remapped recordNumber to new individualCode where applicable
- 08:44 AM Revision 7195: mappings/VegCore-VegBIEN.csv: Mapped individualCode. authortaxoncode: Prefer tag over recordNumber (collectionnumber), because this applies to the plant rather than the specimen.
- 08:17 AM Revision 7194: mappings/VegCore-VegBIEN.csv: Mapped morphoname
- 08:16 AM Revision 7193: mappings/VegCore.csv: Regenerated from wiki
- 08:14 AM Revision 7192: mappings/VegCore.csv: Regenerated from wiki
- 08:04 AM Revision 7191: schemas/vegbien.sql: taxonverbatim: Added morphoname (which is different from the morphospecies suffix)
- 07:33 AM Revision 7190: schemas/vegbien.sql: plantobservation: Renamed collectionnumber to authorplantcode since this number, which identifies the *plant*, is actually different from the collectionnumber that identifies the *specimen* collected from it. This distinction is meaningful for plots data, but generally not for specimens data.
- 07:28 AM Revision 7189: schemas/vegbien.sql: plantobservation: Renamed collectionnumber to authorplantcode since this number, which identifies the *plant*, is actually different from the collectionnumber that identifies the *specimen* collected from it. This distinction is meaningful for plots data, but generally not for specimens data.
- 07:23 AM Revision 7188: schemas/vegbien.sql: specimenreplicate: Added collectionnumber
- 07:17 AM Revision 7187: schemas/vegbien.sql: taxonlabel: Removed no longer used matched_label_fit_fraction. Use taxondetermination.taxonfit instead.
- 07:02 AM Revision 7186: inputs/*/*/test.xml.ref: Restored inserted row counts, which had gotten auto-accepted from a test run on a non-empty DB
- 06:55 AM Revision 7185: schemas/vegbien.ERD.mwb: Expanded analytical_stem to fit the width of all fields
- 06:53 AM Revision 7184: schemas/vegbien.sql: taxondetermination: taxondetermination_computer_min_fit CHECK constraint: Fixed bug where need to use CASE instead of OR when a branch of an OR shouldn't be evaluated, because PostgreSQL doesn't support short-circuit OR
- 06:38 AM Revision 7183: README.TXT: Debugging: Added instructions for "binary chop" debugging, which requires syncing the DB schema to the svn working copy
- 06:08 AM Revision 7182: mappings/VegCore-VegBIEN.csv: Removed no longer used mappings for verbatimScientificName in _if conditions
- 06:08 AM Revision 7181: inputs/.NCBI/nodes/test.xml.ref: Restored inserted row counts, which had gotten auto-accepted from a test run on a non-empty DB
- 06:06 AM Revision 7180: sql_io.py: put_table(): DuplicateKeyException: Uniquifying input table to avoid internal duplicate keys: Also filter out duplicate rows in the out_table, so that they don't create duplicate key errors and the resulting index holes
- 06:01 AM Revision 7179: sql.py: distinct_table(): Added support for custom joins used in creating the new table. This can then be used by sql_io.put_table() to filter out duplicate rows in the out_table, so that they don't create duplicate key errors and the resulting index holes.
- 05:53 AM Revision 7178: README.TXT: Documentation: Redmine-formatted list of steps for column-based import: Added step to reinstall public schema first, to reset the sequences so that they don't create a diff when the new steps.by_col.log.sql is committed
- 05:48 AM Revision 7177: Added inputs/ACAD/Specimen/logs/steps.by_col.log.sql
- 05:45 AM Revision 7176: sql_gen.py: Join: Added support for mapping values which are lists, for use in USING joins
- 05:40 AM Revision 7175: inputs/SALVIAS/*/test.xml.ref: Restored SALVIAS* inserted row counts, which had gotten auto-accepted from a test run on a non-empty DB
- 05:01 AM Revision 7174: schemas/vegbien.sql: analytical_stem: Added locationName (authorPlotCode), subplot, individualCode (authorPlantCode) for use in validation
- 04:57 AM Revision 7173: schemas/vegbien.sql: sync_analytical_stem_to_view(): Drop and re-create dependent objects to avoid errors that analytical_stem can't be dropped because of dependents
- 04:56 AM Revision 7172: schemas/vegbien.sql: sync_analytical_stem_to_view(): Changed to PL/pgSQL function to allow adding PL/pgSQL commands
- 03:26 AM Revision 7171: schemas/vegbien.ERD.mwb: Moved family_higher_plant_group to leave room for analytical_stem to expand
- 03:08 AM Revision 7170: mappings/VegCore-VegBIEN.csv: Removed no longer used mappings for verbatimScientificName in _if conditions
- 02:59 AM Revision 7169: mappings/VegCore-VegBIEN.csv: Removed taxonlabel for original taxondetermination, because the original taxondetermination is not scrubbed by scrub.make (only the most current taxondetermination gets scrubbed, because only a single scrubbed determination is added by scrub.make). This still leaves the original taxondetermination's taxonverbatim, which stores the taxonomic information for historical purposes.
- 02:44 AM Revision 7168: mappings/VegCore-VegBIEN.csv: Removed no longer used accepted and verbatim (parsed) taxonlabels, which have been replaced by a single accepted or matched taxondetermination created by scrub.make
- 02:34 AM Revision 7167: Removed no longer used inputs/.TNRS/tnrs_accepted, tnrs_other. Use the tnrs_canon view instead.
- 02:22 AM Revision 7166: Removed no longer used inputs/.TNRS/tnrs_accepted, tnrs_other. Use the tnrs_canon view instead.
- 02:18 AM Revision 7165: Added inputs/.TNRS/_archive/
- 02:18 AM Revision 7164: Added inputs/.TNRS/tnrs/cleanup.sql to prevent running the default cleanup operations, which don't work on tables which have views referencing them (as is the case for tnrs, which is referenced by tnrs_canon)
- 02:07 AM Revision 7163: import_all: Removed no longer needed TNRS import, which has been replaced by scrub.make (which adds TNRS taxondeterminations after the import instead of creating taxonlabel links before it)
- 02:03 AM Revision 7162: mappings/VegCore-VegBIEN.csv: Removed TNRS input taxonlabels meant to cross-link to taxonlabels added by the TNRS import, because TNRS taxondeterminations are now created instead
- 01:42 AM Revision 7161: schemas/vegbien.sql: analytical_stem_view: Use just the main taxonlabel created by scrub.make instead of all the additional taxonlabels created by the TNRS import
- 01:11 AM Revision 7160: mappings/VegCore-VegBIEN.csv: main taxonverbatim.morphospecies "if has verbatim name" condition: Fixed bug where need to remove the taxonIsCanonical flag, because the TNRS.public.unscrubbed_taxondetermination_view table (which uses this flag) *should* include this field (although not other places where the morphospecies is stored by other TNRS tables)
- 12:49 AM Revision 7159: schemas/vegbien.sql: taxondetermination: taxondetermination_set_iscurrent() trigger: Also run on delete, to mark another taxondetermination as the current one when a current taxondetermination is deleted
- 12:18 AM Revision 7158: inputs/.TNRS/schema.sql: tnrs_canon: Annotations: Always use value from the matched name, because the accepted name does not have this
- 12:05 AM Revision 7157: mappings/VegCore-VegBIEN.csv: primary taxonlabel's parent taxonlabel: Fixed bug where a taxonverbatim was incorrectly being created solely to store the taxonRank, even though it was already stored in the taxonlabel's rank field
01/10/2013
- 11:52 PM Revision 7156: mappings/VegCore-VegBIEN.csv: Don't map morphospecies to the parsed taxonlabel's taxonepithet, because this causes an extra, parsed taxonlabel to be created for TNRS.public.unscrubbed_taxondetermination_view. It is not needed by the other TNRS tables.
- 11:45 PM Revision 7155: inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Omit Infraspecific_rank to help avoid creating a separate, parsed taxonlabel. Don't map to taxonRank because Name_matched_rank is populated more often.
- 11:34 PM Revision 7154: inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Reduced $maxPause to 4 hr, because new taxondeterminations are being added throughout the import, so it is unlikely that more than more than 4 hr would pass between successive imports of taxondeterminations (causing scrub.make to stop prematurely)
- 11:23 PM Revision 7153: inputs/.TNRS/schema.sql: Removed no longer used tnrs+accepted. Use tnrs_canon or a self-join of tnrs instead
- 11:22 PM Revision 7152: schemas/vegbien.sql: tnrs_input_name: Use TNRS.tnrs directly instead of the now-deprecated tnrs+accepted
- 11:12 PM Revision 7151: schemas/vegbien.sql: Use new TNRS.tnrs_canon instead of tnrs+accepted to avoid creating additional taxonlabels for the parsed, matched, and accepted names and instead just use the most-canonicalized name of the names output by TNRS (the accepted name if available, or the matched name otherwise)
- 10:50 PM Revision 7150: mappings/VegCore-VegBIEN.csv: "if has verbatim name" _if statements that filter something out for TNRS mappings: Also assume true if taxonIsCanonical is specified, because some TNRS tables (eventually such as public.unscrubbed_taxondetermination_view) do not specify a separate "verbatim" taxondetermination but do provide taxonIsCanonical as a flag to turn various mappings on and off
- 09:06 PM Revision 7149: mappings/VegCore-VegBIEN.csv: Remapped matched*Fit_fraction to taxondetermination.taxonfit when a taxondetermination, not just a taxonlabel, is provided
- 09:03 PM Revision 7148: bin/map: map_table(): Resolving prefixes: Fixed bug where need to use list instead of tuple for metadata value mappings
- 08:16 PM Revision 7147: schemas/vegbien.sql: taxondetermination: Added CHECK constraint to allow only taxondeterminations with a minimum fit fraction of 80%, analogous to taxonlabel's taxonlabel_1_matched_label_min_fit() trigger
01/09/2013
- 05:34 PM Revision 7146: mappings/VegCore-VegBIEN.csv: Don't create a separate TNRS input taxonlabel if taxonIsCanonical exists
- 05:24 PM Revision 7145: inputs/.TNRS/schema.sql: tnrs_canon: Fixed bug where need to always use Unmatched_terms from tnrs rather than tnrs_accepted
- 05:07 PM Revision 7144: inputs/.TNRS/schema.sql: Added tnrs_canon, which stores the most canonicalized name output by TNRS
- 04:17 PM Revision 7143: schemas/vegbien.sql: analytical_stem_view: accepted_taxonverbatim: Fixed bug where need to join only to the taxonverbatim whose morphospecies is NULL, to avoid joining to multiple taxonverbatims at once. This extra filter is now needed because there can be multiple taxonverbatims for a taxonlabel with different morphospecies.
- 03:59 PM Revision 7142: mappings/VegCore-VegBIEN.csv: taxonlabel.taxonomicname: Prepend the family to the rest of the name using new _merge_prefix() instead of _join_words()/_nullIf(), so that any input taxonomic name that includes the family will not have the family duplicated in the combined taxonomic name. Previously, the duplication was removed only when the rest of the input name was *equal to* the family. This change fixes a bug in the new TNRS import where a pre-concatenated taxonomic name (Accepted_scientific_name) which includes the family is now used instead of Accepted_name, which only includes it when it's equal to the family.
- 03:52 PM Revision 7141: xml_func.py: Simplifying functions: Merging: Added _merge_prefix() passthru
- 03:33 PM Revision 7140: schemas/functions.sql: Added _merge_prefix()
- 02:42 PM Revision 7139: inputs/.TNRS/schema.sql: tnrs_populate_accepted_scientific_name(): Fixed bug where Accepted_name_family shouldn't be prefixed to Accepted_name if Accepted_name is itself the family, to avoid duplicating the family in the Accepted_scientific_name
- 02:18 PM Revision 7138: inputs/.TNRS/schema.sql: tnrs+accepted: Added new Accepted_scientific_name column and mapped it in public.unscrubbed_taxondetermination_view
- 11:06 AM Revision 7137: schemas/vegbien.sql: tnrs_input_name: Fixed bug where need to filter out tnrs+accepted rows with NULL Accepted_scientific_name, because inputs to tnrs_db must be strings
- 10:53 AM Revision 7136: schemas/vegbien.sql: tnrs_input_name: Prepend TNRS accepted names that have not yet been parsed. This allows parsing TNRS accepted names without first needing to import them into taxonlabels, which may not occur until the next import.
- 10:09 AM Revision 7135: inputs/.TNRS/schema.sql: tnrs+accepted: Use new Accepted_scientific_name to join to tnrs_accepted.Name_submitted
- 10:05 AM Revision 7134: inputs/.TNRS/schema.sql: tnrs: Added tnrs_populate_accepted_scientific_name() trigger
- 09:57 AM Revision 7133: inputs/.TNRS/schema.sql: tnrs: Added Accepted_scientific_name field which will contain the joined-together accepted name that gets re-parsed by TNRS
- 09:13 AM Revision 7132: inputs/.TNRS/: Changed tnrs+accepted to a view (defined in schema.sql) so accepted names would automatically be populated as they are parsed by TNRS, rather than needing to run `make inputs/.TNRS/tnrs+accepted/reinstall` to populate them
- 08:16 AM Revision 7131: mappings/VegCore-VegBIEN.csv: Also map the morphospecies to the accepted taxonverbatim when an accepted name is provided
- 08:01 AM Revision 7130: schemas/vegbien.sql: taxonverbatim: taxonverbatim_unique: Added morphospecies so that there can be multiple taxonverbatims for the same taxonlabel, each with different morphospecies suffixes
- 04:17 AM Revision 7129: inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Mapped Accepted_name.*
- 03:02 AM Revision 7128: schemas/vegbien.sql: unscrubbed_taxondetermination_view: Use new tnrs+accepted instead of tnrs so that the accepted name can be imported at the same time
- 02:23 AM Revision 7127: import_all: Reinstall tnrs+accepted, for eventual use by unscrubbed_taxondetermination_view
- 02:20 AM Revision 7126: Added inputs/.TNRS/tnrs+accepted/, which self-joins the TNRS results to their parsed accepted names
- 02:02 AM Revision 7125: import_all: Directly import just the TNRS tables that should be imported, because some TNRS tables are included in import_order.txt so that they are part of the automated testing, but should not be imported at the same time as tnrs_accepted/tnrs_other
- 12:45 AM Revision 7124: inputs/import.stats.xls: Updated import times
01/08/2013
- 11:24 PM Revision 7123: with_all: $all mode: Fixed bug where need " " before # for it to be interpreted as a comment (unlike in a Makefile, where the " " often needs to be left out to avoid it being treated as part of a variable value)
- 10:55 PM Revision 7122: bin/map: Made $redo flag default to off, because redo mode is slow (all tables have to be truncated) and is only needed when running tests on a public schema with data in it, which would not be the case on a development machine where tests are usually run
- 10:19 PM Revision 7121: import_all: Made temporary vars local, so they wouldn't affect the calling shell
- 09:45 PM Revision 7120: schemas/vegbien.sql: unscrubbed_taxondetermination_view: Sort by taxondetermination.taxonoccurrence_id instead of taxondetermination_id to allow scanning the taxondetermination_single_current_determination index, which includes only current determinations and avoids needing to scan past many non-current determinations. Note that using taxonoccurrence_id does not create sort order ambiguity between taxondeterminations with the same taxonoccurrence_id, because there is only one current determination per taxonoccurrence.
- 09:32 PM Revision 7119: schemas/vegbien.sql: unscrubbed_taxondetermination_view: Inner-join to taxonverbatim and taxonlabel instead of LEFT JOINing, because only taxondeterminations with a taxonlabel can have accepted taxondeterminations (otherwise there would be no name to scrub)
- 09:30 PM Revision 7118: schemas/vegbien.sql: unscrubbed_taxondetermination_view: Inner-join to tnrs instead of LEFT JOINing, because only taxondeterminations whose taxonlabels have already been scrubbed by TNRS should have accepted taxondeterminations added. Removed now-unneeded filter by tnrs.Name_submitted IS NOT NULL, which is replaced by the inner join.
- 08:46 PM Revision 7117: sql_io.py: put_table(): ensure_cond(): Fixed bug where need to wrap strings used in the tracked error message in strings.ustr()
- 08:33 PM Revision 7116: xml_dom.py: replace_with_text(): Fixed bug where need to use scalar.is_nonnull_scalar() instead of is_scalar() to avoid converting None values to the string 'None'
- 08:32 PM Revision 7115: scalar.py: Added is_nonnull_scalar()
01/07/2013
- 08:17 PM Revision 7114: README.TXT: Data import: Fixed bug where `make inputs/upload` needs to be run on local machine, not vegbiendev
- 08:16 PM Revision 7113: sql.py: create_table(): Support creating a table like a view
- 08:04 PM Revision 7112: sql.py: Added InvalidTypeException and parse it in parse_exception()
- 07:39 PM Revision 7111: mappings/VegCore.csv: Regenerated from wiki
- 07:34 PM Revision 7110: schemas/vegbien.sql: taxondetermination_set_iscurrent(): Fixed bug where need to sort scrubbed determinations first for scrub.make to work. (Otherwise, a datasource determination might remain iscurrent even after a scrubbed determination was added, causing scrub.make to repeatedly attempt to re-add it.)
- 07:20 PM Revision 7109: inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Set dateIdentified to _now()
- 07:20 PM Revision 7108: inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Unset $n to avoid limiting the # rows/iteration
- 07:15 PM Revision 7107: schemas/py_functions.sql: parse_date_range(): Don't parse strings containing a time, because - and ' ' don't have the same meaning as in a date range
- 07:03 PM Revision 7106: xml_dom.py: replace_with_text(): Support any scalar type recognized by scalar.is_scalar()
- 06:54 PM Revision 7105: scalar.py: is_scalar(): Added datetime.datetime
- 06:43 PM Revision 7104: schemas/functions.sql: Added _now()
- 06:39 PM Revision 7103: import_all: Make $dump_opts, $public_import local vars, so they will be automatically unset if the script is aborted
- 06:31 PM Revision 7102: mappings/VegCore-VegBIEN.csv: identificationType: Fixed bug in mapping where extra *_id/ needed to be removed
- 06:25 PM Revision 7101: inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Set taxonOccurrenceID to dummy value 0 to enable the taxonoccurrence CHECK constraint to pass. This is needed because the constraint must pass before the pkey (which should already exist) is even checked.
- 06:19 PM Revision 7100: inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Set identificationType to computer
- 06:18 PM Revision 7099: mappings/VegCore-VegBIEN.csv: Mapped identificationType
- 06:15 PM Revision 7098: mappings/VegCore.csv: Regenerated from wiki
- 05:39 PM Revision 7097: schemas/vegbien.sql: unscrubbed_taxondetermination_view: Use `SELECT source_id FROM source WHERE shortname = ...` instead of source_by_shortname() so that the source table is updated to point to the same schema as the view rather than pointing to whichever version (usually public) is first in the search_path
- 05:23 PM Revision 7096: schemas/vegbien.sql: unscrubbed_taxondetermination_view: Fixed bug where need to include only those taxondeterminations that already have a match in TNRS.tnrs, to avoid adding empty TNRS taxondeterminations. As the concurrent tnrs daemon runs, these taxondeterminations will gradually acquire matches in tnrs and then be processed by scrub.
- 05:00 PM Revision 7095: import_all: Make $import_source a local var, so it will be automatically unset if the script is aborted
- 04:49 PM Revision 7094: vegbien_dest: Schema override for referring to a table in the $public schema: Only process the override when $!schemaVar and $!tableVar are non-*empty*, to allow setting $schema=""
- 04:47 PM Revision 7093: schemas/Makefile: DDL generation: vegbien.sql: Unset $dump_opts so that pg_dump does not use env vars left after running import_all
- 04:44 PM Revision 7092: schemas/Makefile: DDL generation: vegbien.sql: Unset $version so that pg_dump always uses the public schema, even after running import_all
- 04:13 PM Revision 7091: mappings/VegCore.csv: Regenerated from wiki
- 04:13 PM Revision 7090: README.TXT: Testing: Added commands to put in .profile on a development machine
- 04:10 PM Revision 7089: import_all: Added command to add scrubbed taxondeterminations
- 04:09 PM Revision 7088: mappings/VegCore.csv: Regenerated from wiki
- 04:08 PM Revision 7087: import_all: Start tnrs-remake *after* starting the inputs, so that for subset imports (e.g. n=2), there will already be names to scrub when tnrs-remake starts up and it won't enter pause mode to wait for new rows (the pause is calibrated for full imports, and is too long for subset imports)
- 04:01 PM Revision 7086: with_all: Also exclude .archive/ from the subdirs to forward commands to
- 03:40 PM Revision 7085: inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Added option to wait for new rows, in the same way tnrs_db does
- 03:38 PM Revision 7084: inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make: Factored new rows added test out into rowsAdded() function
- 03:09 PM Revision 7083: Added inputs/.TNRS/public.unscrubbed_taxondetermination_view/scrub.make, which adds scrubbed taxondeterminations to VegBIEN
- 02:00 PM Revision 7082: root Makefile: Removed $(subMake), which is now defined properly by lib/common.Makefile
- 01:59 PM Revision 7081: lib/common.Makefile: $(subMake): Removed `--makefile=../input.Makefile`, which is specific just to inputs/Makefile
- 01:43 PM Revision 7080: input.Makefile: Import to VegBIEN: $(import): Print the date at the beginning of the import, so successive imports to the same version can be distinguished
- 01:37 PM Revision 7079: input.Makefile: Import to VegBIEN: \$(import): Fixed bug where 2>&1 needs to come after >>$(log_) rather than before
- 01:35 PM Revision 7078: inputs/.TNRS/tnrs/tnrs.make: Usage: Added tnrs_db's $wait flag
- 01:34 PM Revision 7077: inputs/.TNRS/tnrs/tnrs.make: Fixed Usage message to use make, which is needed to set the PATH correctly
- 11:47 AM Revision 7076: Makefiles: Changed "Usage: `make -s ...`" to "Run with `make -s` to avoid echoing make commands"
- 11:44 AM Revision 7075: input.Makefile: Import to VegBIEN: Added %/log_file to view the import log file path
- 11:28 AM Revision 7074: input.Makefile: Import to VegBIEN: $(import): Append to the log file instead of replacing it, to avoid overwriting the log for a previous import to the same versioned schema. This allows a datasource to be (re-)imported multiple times, and is needed by the new method for linking taxonoccurrences to scrubbed taxonomic names.
- 11:22 AM Revision 7073: input.Makefile: Import to VegBIEN: $(import): Always output just to log file if $(log) is on, rather than also copying output to the terminal when $(n) is set. When $(log) is on, the output can still be viewed by tailing the log.
- 11:16 AM Revision 7072: input.Makefile: Import to VegBIEN: $(import): Merged consecutive $(if $(n),...)
- 11:14 AM Revision 7071: input.Makefile: Import to VegBIEN: $(import): Merged consecutive $(if $(log),...)
- 11:07 AM Revision 7070: Added inputs/.TNRS/public.unscrubbed_taxondetermination_view/
- 11:05 AM Revision 7069: mappings/VegCore-VegBIEN.csv: Mapped taxonOccurrencePkey
- 10:58 AM Revision 7068: mappings/VegCore.csv: Regenerated from wiki
- 10:41 AM Revision 7067: input.Makefile: Staging tables installation: Added %_view/install, to prevent trying to edit a view during installation
- 10:31 AM Revision 7066: vegbien_dest: Added schema override support for referring to a table in the $public schema
- 10:29 AM Revision 7065: input.Makefile: Staging tables installation: $(cleanup): Moved setting of $schema, $table before vegbien_dest is run, so it can modify them if needed
- 09:42 AM Revision 7064: inputs/.TNRS/tnrs/tnrs.make: Removed unnecessary setting of $prefix, which now defaults to ""
- 09:40 AM Revision 7063: schemas/vegbien.sql: Added unscrubbed_taxondetermination_view
01/04/2013
- 10:10 PM Revision 7062: inputs/import.stats.xls: Moved CTFS to Deleted section
- 10:03 PM Revision 7061: make_analytical_db: ANALYZE each table after its created so that queries use index scans instead of seq scans
- 09:40 PM Revision 7060: schemas/vegbien.sql: sync_analytical_*_to_view(): Added datasource fkey to source.shortname so removing a datasource will also remove the corresponding rows in the analytical views
- 09:36 PM Revision 7059: schemas/vegbien.sql: Regenerated analytical_stem using sync_analytical_stem_to_view()
- 09:19 PM Revision 7058: input.Makefile: Editing import: rm: Time the command
- 09:15 PM Task #549 (Resolved): add covering indexes on fkeys
- 07:09 PM Task #549: add covering indexes on fkeys
- Added for direct fkeys which are in use
- 05:56 PM Task #549 (Resolved): add covering indexes on fkeys
- * use instructions for *[[Postgres queries#Adding covering indexes on foreign keys|adding covering indexes on foreign...
- 09:14 PM Revision 7057: schemas/vegbien.sql: Added covering indexes where needed, as described at <https://projects.nceas.ucsb.edu/nceas/issues/549>
- 09:11 PM Revision 7056: schemas/vegbien.sql: Fixed fkey constraint names
- 09:09 PM Revision 7055: schemas/vegbien.sql: Added covering indexes where needed, as described at <https://projects.nceas.ucsb.edu/nceas/issues/549>
- 06:59 PM Revision 7054: schemas/vegbien.sql: fkeys to source: Added covering indexes where needed, as described at <https://projects.nceas.ucsb.edu/nceas/issues/549>
- 06:22 PM Revision 7053: schemas/vegbien.sql: commconcept: Renamed source_id back to reference_id (it was previously renamed to source_id in a bulk rename)
- 06:20 PM Revision 7052: schemas/vegbien.sql: taxondetermination: Added back reference_id, which is different than the scoping source_id (reference_id was previously renamed to source_id in a bulk rename)
- 06:04 PM Revision 7051: schemas/vegbien.sql: Renamed taxonconcept_concept_source_id_fkey back to taxonconcept_concept_reference_id_fkey
- 06:02 PM Revision 7050: schemas/vegbien.sql: Renamed *_reference_id_fkey fkeys to *_source_id_fkey
- 05:32 PM Revision 7049: inputs/CTFS/_no_import: Temporarily remove CTFS from the public DB per Rick Condit's request (due to validation issues)
- 05:25 PM Revision 7048: import_all: Run import with $public_import set in order to exclude excluded datasources
- 05:23 PM Revision 7047: input.Makefile: Import to VegBIEN: %/import: Don't run the import if $public_import flag is set and the datasource contains a _no_import file. This allows just excluding a datasource from the public DB, without also removing it from automated testing.
- 05:17 PM Revision 7046: lib/common.Makefile: Added $(and), $(or), $(not)
- 04:30 PM Revision 7045: mappings/VegCore.csv: Regenerated from wiki
- 04:12 PM Revision 7044: schemas/vegbien.sql: taxondetermination: Added scoping source_id field to allow other datasources (e.g. TNRS) to make taxondeterminations. (Repurposed existing non-scoping source_id.)
01/03/2013
- 08:25 AM Revision 7043: make_analytical_db: Fixed bug where can't give public_ select access to all analytical_db views because this apparently adds access rather than passing through the underlying table's permissions
- 08:18 AM Revision 7042: make_analytical_db: Give public_ select access to analytical_db views. This causes the actual access to depend on the underlying table's permissions.
- 07:43 AM Revision 7041: mappings/VegCore.csv: Regenerated from wiki
- 07:39 AM Revision 7040: mappings/Makefile: VegCore.csv: Include only terms that start with a lowercase letter or are all caps. This also avoids the need to filter out VegCore.tables.csv.
- 07:31 AM Revision 7039: mappings/VegCore.csv: Changed line endings to \n in preparation for not running filter_out_cs on the file (which changes line endings to \r\n)
- 02:31 AM Revision 7038: import_all: `make backups/vegbien.$version.backup/test`: Documented that this uses $dump_opts. $dump_opts must be manually set when running this command outside of import_all.
- 02:21 AM Revision 7037: backups/Makefile: Synchronization: %/download: Download the .md5 file first, so that the user is prompted right away for their password rather than after the main file has finished downloading, by which time the password prompt has timed out
- 12:02 AM Revision 7036: mappings/VegCore.csv: Regenerated from wiki
01/02/2013
- 11:34 PM Revision 7035: mappings/VegCore.csv: Regenerated from wiki
- 11:32 PM Revision 7034: mappings/Makefile: VegCore.csv: Fixed bug where need to filter out VegCore.tables.csv case-sensitively so that field names which are the same as a table name don't get filtered out
- 11:23 PM Revision 7033: Added filter_out_cs
- 09:21 PM Revision 7032: README.TXT: Data import: Added step to ensure there are no local modifications using `svn st`
- 08:56 PM Revision 7031: make_analytical_db: Also grant USAGE on the analytical_db schema itself to bien_read, public_
- 07:14 PM Revision 7030: README.TXT: Data import: After import: Also check that the provider_count table contains entries for all inputs
- 07:03 PM Revision 7029: Added inputs/.geoscrub/_src/geovalidity-table.txt, which was attached to Jim's geovalidation e-mail (provided in README.TXT)
- 06:43 PM Revision 7028: inputs/import.stats.xls: Updated import times
- 06:39 PM Revision 7027: README.TXT: Data import: recording the import times in inputs/import.stats.xls: Updated column group header to "By column"
- 06:36 PM Revision 7026: backups/Makefile: Removed no longer used $(psqlVerbose)
- 06:36 PM Revision 7025: backups/Makefile: Removed %.backup/rm_indexes, which is no longer needed because archived imports are now backed up instead of being stored without indexes in the live DB
- 06:31 PM Revision 7024: backups/Makefile: %.backup/remove: Fixed bug where need to use $no_search_path option to psql_script_vegbien
12/21/2012
- 03:34 PM Revision 7023: import_all: Allow caller to override $dump_opts
- 03:33 PM Revision 7022: pg_dump_vegbien: Renamed $opts env var to $dump_opts to avoid conflicting with other commands' vars of the same name
- 03:22 PM Revision 7021: mappings/VegCore.csv: Regenerated from wiki
- 03:20 PM Revision 7020: mappings/VegCore.csv: Regenerated from wiki
- 02:14 PM Revision 7019: schemas/vegbien.sql: location: Removed location_unique_within_parent_by_sourceaccessioncode, which duplicates location_unique_within_creator_by_sourceaccessioncode because the sourceaccessioncode is globally unique
- 02:10 PM Revision 7018: schemas/vegbien.sql: analytical_stem_view: projectID: Use project.projectname if project.sourceaccessioncode isn't provided
- 02:02 PM Revision 7017: schemas/vegbien.sql: location: location_unique_within_parent: Split into *_by_sourceaccessioncode and *_by_authorlocationcode_position, with each ID being matched separately. This way, if the initial import of a subplot's location provides both fields, but fkey references use only one field, the fkey references will still match the existing location because only one of the fields needs to match.
- 01:26 PM Revision 7016: schemas/vegbien.sql: analytical_stem_view: elevationInMeters: Use parent_location.elevation_m when location.elevation_m not provided
- 01:17 PM Revision 7015: schemas/vegbien.sql: analytical_stem_view: scientificName: Fixed bug where need to use accepted_taxon*label*.taxonomicname instead of accepted_taxonverbatim.taxonomicname, because taxonverbatim's name component fields aren't populated if the name doesn't match a scrubbed name. The datasource's own taxonverbatim can't be used for this because the canon_label_id refers to the concatenated taxonomic name owned by the TNRS datasource.
- 01:00 PM Revision 7014: inputs/NVS/Plot/map.csv: Corrected Plot ID mapping to go to subplotID instead of locationID, because each subplot gets its own ID in this field
- 12:50 PM Revision 7013: schemas/vegbien.sql: location: location_unique_within_parent: Also apply this constraint when sourceaccessioncode is provided, because it may be a concatenated value populated for use by the analytical DB but which is not used as an fkey by the datasource itself
- 12:30 PM Revision 7012: schemas/vegbien.sql: analytical_stem_view: locationID: Concatenate parent location's and subplot's IDs using '; ' instead of ' '
- 12:22 PM Revision 7011: schemas/vegbien.sql: analytical_*: Renamed locationName to locationID because it's now globally unique (within the datasource) and can be used as a sourceaccessioncode
- 12:19 PM Revision 7010: schemas/vegbien.sql: analytical_stem_view: locationName: For subplots without their own sourceaccessioncode (globally unique ID), prepend the parent location's unique ID so that locationName is globally unique
- 12:07 PM Revision 7009: mappings/VegCore-VegBIEN.csv: locationID/locationName + subplot -> location.sourceaccessioncode mapping: Fixed bug where subplot was incorrectly being mapped to this field even when there was no location*. (This field can only be populated if both location* *and* subplot are specified.) Also only map locationID for this, to avoid inconsistencies where one table supplies locationID+subplot, while another table supplies locationName+subplot, but they both get mapped to the same field, preventing plots from being matched up with their observations when creating the analytical_stem.
- 11:31 AM Revision 7008: xml_func.py: Simplifying functions: Logic: _and(), _or(): Evaluate an expression of only constant values
- 11:30 AM Revision 7007: lists.py: Added and_(), or_()
- 11:28 AM Revision 7006: xml_func.py: is_scalar(): Fixed bug where need to check if value is a string before calling is_var_name()
- 10:15 AM Revision 7005: inputs/NVS/StemObservation/map.csv: Remapped Verbatim Code to authorTaxonCode, because as it's used this is actually an identifier for the taxon, not the stem, despite Nick Spencer's revised mapping
12/20/2012
- 05:21 PM Revision 7004: schemas/vegbien.ERD.mwb: Regenerated exports
- 05:21 PM Revision 7003: README.TXT: Schema changes: Update graphical ERD exports: Added step to commit changes
- 05:02 PM Revision 7002: inputs/NVS/*/map.csv: Remapped with Nick Spencer's suggested changes
- 04:41 PM Revision 7001: xml_func.py: _first(): Fixed bug where need to choose the first *non-empty* param, by first pruning empty child nodes
- 04:38 PM Revision 7000: mappings/VegCore-VegBIEN.csv: authortaxoncode mappings: Only using authorTaxonCode if there is no plant ID: Added individualID, stemID to the terms that cause authorTaxonCode not to be mapped to VegBIEN authortaxoncode
- 04:03 PM Revision 6999: mappings/VegCore-VegBIEN.csv: authortaxoncode mappings: Only using authorTaxonCode if there is no plant ID: Added individualID, stemID to the terms that cause authorTaxonCode not to be mapped to VegBIEN authortaxoncode
- 03:59 PM Revision 6998: schemas/vegbien.sql: analytical_*: Renamed individualID to individualObservationID because this actually corresponds to plantobservation.sourceaccessioncode, which is an observation *of* an individual
- 03:56 PM Revision 6997: mappings/VegCore.csv: Regenerated from wiki
- 03:53 PM Revision 6996: README.TXT: Data import: Recording the import times: Changed <version> back to $version because these commands are actually run on vegbiendev, where $version is set. (Modifications to import.stats.xls would be made on your local machine.)
- 03:50 PM Revision 6995: README.TXT: Data import: Added step to unset $version before starting the import, to avoid importing on top of the last import's data
- 02:47 PM Revision 6994: README.TXT: Data import: Replaced $version with <version> where it needs to be manually filled in
- 02:40 PM Revision 6993: README.TXT: Data import: On nimoy: Added command to set $version
- 02:36 PM Task #548 (New): remove benign errors from the data provider feedback tables
- h3. Examples...
- 02:26 PM Revision 6992: mappings/VegCore-VegBIEN.csv: authortaxoncode mappings: Only use authorTaxonCode if there is no plant ID, because an individual plant gets its own taxonoccurrence and thus needs the taxonoccurrence's IDs to be unique to the plant, regardless of what the author designates as the taxonoccurrence code
- 01:47 PM Revision 6991: Generated inputs/NVS/new_terms.csv
- 01:47 PM Revision 6990: input.Makefile: SVN: $(svnFilesGlob): Also match *terms.csv in top-level dir
- 01:23 PM Revision 6989: mappings/VegCore-VegBIEN.csv: Mapped authorTaxonCode
- 01:12 PM Revision 6988: mappings/VegCore.csv: Regenerated from wiki
- 01:12 PM Revision 6987: README.TXT: Maintenance: VegCore data dictionary: Added step to commit updated mappings/VegCore.csv
- 12:13 PM Revision 6986: schemas/Makefile: %/publish: Fixed bug where commands were not being run transactionally, because --single-transaction requires `--file -` to work properly
- 11:36 AM Revision 6985: input.Makefile: Editing import: Removed rotate because appending the current svn revision doesn't make sense, since this is not related to the revision used to import the datasource
- 11:34 AM Revision 6984: input.Makefile: Editing import: Added rename/% and use it in rotate
- 11:21 AM Revision 6983: inputs/import.stats.xls: Updated import times
- 11:21 AM Revision 6982: schemas/Makefile: Use $* instead of $(@D) for clarity. $(@D) is only needed when the dir part of the target includes a prefix in addition to the % stem.
- 10:45 AM Revision 6981: make_analytical_db: Automatically call export_analytical_db when finished
- 10:35 AM Revision 6980: schemas/vegbien.sql: make_family_higher_plant_group(): Added `taxonepithet IS NOT NULL` filter, to allow make_analytical_db to proceed even when the NCBI import fails (leaving some nodes with rank = 'family' but no associated taxonepithet). The most recent NCBI import failed due to the search_path/DuplicateException bug resulting from the import schema and public being in the search_path together.
- 10:14 AM Revision 6979: schemas/Makefile: Fixed bug where need `SHELL := /bin/bash` for \$(confirmRmPublicSchema) to work correctly
- 10:12 AM Revision 6978: lib/common.Makefile: $(confirm): Added comment that this requires `SHELL := /bin/bash` to work correctly
- 10:09 AM Revision 6977: import_all: after_import(): Added `make backups/vegbien.$version.backup/test`
- 10:05 AM Revision 6976: sql.py: DbConn._db(): search_path: Don't append the existing search_path, because it usually includes the public schema, which is now different from the schema being imported into. This fixes a bug where sql.function_exists() would find public-schema functions in both the public schema and the import's schema because both were in the search_path, causing a DuplicateException "more than one function named ...". Note that the elements of the existing search_path are no longer needed now that vegbien_dest's $schemas includes $public. Also note that if an instance of DbConn does not specify the schemas param, the existing search_path will be left as-is rather than overwritten with an empty list.
- 09:54 AM Revision 6975: README.TXT: Data import: recording the import times in inputs/import.stats.xls: Added step to determine the import date using import_date
- 09:52 AM Revision 6974: import_date: Added note that Mac and Linux differ in the order they sort the logs in
- 09:50 AM Revision 6973: README.TXT: Data import: recording the import times in inputs/import.stats.xls: Updated pattern for new log filename format
- 09:47 AM Revision 6972: README.TXT: Data import: recording the import times in inputs/import.stats.xls: Removed extra ./ before bin/import_times
- 09:46 AM Revision 6971: import_date: Added note that the time this outputs is the time the first special input *finished* importing. The import itself generally starts a few minutes before that, and the exact time is in that import's public schema comment.
- 09:41 AM Revision 6970: import_date: Removed duplicate Usage message at top of file, which is repeated in the Usage message provided when the program is run with no arguments
- 09:40 AM Revision 6969: Added import_date
- 09:38 AM Revision 6968: Added mtime
- 09:29 AM Revision 6967: lib/common.Makefile: System: Added $(mtime)
- 09:27 AM Revision 6966: lib/common.Makefile: $(date): Factored date format out into $(dateFmt)
- 09:25 AM Revision 6965: backups/Makefile: Factored $(isMac) out into lib/common.Makefile
- 08:51 AM Task #446: fix deadlock in INSERT IGNORE replacement
- Added additional bug occurrence
- 08:30 AM Revision 6964: README.TXT: Data import: tailing logs: Updated pattern for new log filename format
12/19/2012
- 02:02 PM Revision 6963: schemas/Makefile: Installation: %/publish: Fixed bug where need quotes around source schema name
- 01:57 PM Revision 6962: README.TXT: Data import: Moved deletion previous imports before the import, so that full DB backup can be automated
- 01:55 PM Revision 6961: README.TXT: Data import: `make backups/vegbien.$version.backup/test`: Added --exclude-schema=public to leave out the previous (now published) import so it doesn't bloat the backup. Note that public is included in the vegbien.$version.backup for the previous import, named according to its version.
- 01:49 PM Revision 6960: import_all: after_import(): Added `make backups/TNRS.backup-remake`
- 01:46 PM Revision 6959: README.TXT: Data import: Added step to publish the import to the public schema
- 01:42 PM Revision 6958: import_all: after_import(): Added export_analytical_db
- 01:36 PM Revision 6957: README.TXT: Data import: bin/export_analytical_db: Removed `env public=$version` because export_analytical_db now uses $version as $public when provided
- 01:35 PM Revision 6956: README.TXT: Data import: To remake analytical DB: Removed `env public=...` because $version (which replaces $public) is now set automatically by import_all
- 01:32 PM Revision 6955: schemas/Makefile: Installation: py_functions/install: Removed `env public=`, which is not needed since $(psqlAsAdminVegbien) does not use psql_script_vegbien (which uses $public)
- 01:28 PM Revision 6954: export_analytical_db: Use vegbien_dest to set the default value for $public
- 01:21 PM Revision 6953: README.TXT: Data import: If many inputs have errors: Updated command to `make schemas/$version/uninstall` because the current import's schema is now named $version
- 01:15 PM Revision 6952: schemas/Makefile: Installation: $(schemas), $(schemasReversed) (used e.g. by `make schemas/reinstall`): Removed public so that when `make schemas/reinstall` is run before an import, it will not remove any active (published) import which resides in the public schema
- 01:10 PM Revision 6951: README.TXT: Schema changes: Reinstall public separately from the other schemas so that it will still be reinstalled when schemas/reinstall excludes the public schema to avoid removing any active (published) import
- 01:01 PM Revision 6950: vegbien_dest callers: Removed no longer needed explicit setting $prefix to "", because this is now the default value
- 01:00 PM Revision 6949: vegbien_dest: Changed default $prefix to "", so that the majority of callers don't need to manually set $prefix to "" to avoid it defaulting to out_
- 12:45 PM Revision 6948: README.TXT: Data import: Use env var $version, which is now set by import_all, instead of manually inserting the version for <version>
- 12:40 PM Revision 6947: vegbien_dest: Also export $version
- 12:30 PM Revision 6946: import_all: Run the import directly into a new, already-versioned public schema. This removes the need to manually rename the schema after import, and allows the backup commands to use the stored $version shell variable to refer to the last import.
- 12:25 PM Revision 6945: schemas/Makefile: %/publish: Added instruction to run `unset version` after the command, to clear the $version shell variable which will be set by import_all
- 12:12 PM Revision 6944: README.TXT: Data import: Replaced <import_name> with <version> because the import name is now just the version
- 12:10 PM Revision 6943: README.TXT: Data import: Replaced r<revision> with <version> because the version string is now equal to r<revision>
- 12:09 PM Revision 6942: README.TXT: Backups: Replaced <date> with <version> because the date is no longer included in the version string
- 12:08 PM Revision 6941: README.TXT: Name archived imports without the "public." prefix so that their backups will work with the new `make backups/%.backup/remove` command, which does not add back the prefix
- 11:56 AM Revision 6940: backups/Makefile; $(public*): Don't add a "public." prefix to get the name of the public schema
- 11:40 AM Revision 6939: backups/Makefile: Removed no longer used $(rmSchema)
- 11:39 AM Revision 6938: backups/Makefile: Use \$(rmSchemaCmd) from lib/common.Makefile instead of \$(rmSchema)
- 11:20 AM Revision 6937: vegbien_dest: Use $version as $public when $public not provided. When neither is provided, continue to use "public" and also set $version to that.
- 11:11 AM Revision 6936: schemas/Makefile: Installation: rotate: Use just the version, without the "public." prefix
- 11:04 AM Revision 6935: schemas/Makefile: Installation: `public/install public%/install`: Generalized to %/install to allow public schema versions with any name. This requires moving `%/install: %.sql` before it to override it.
- 11:00 AM Revision 6934: schemas/Makefile: Installation: Merged public/install and public%/install
- 10:54 AM Revision 6933: schemas/Makefile: Installation: Moved %/uninstall to beginning of section because it applies to all schemas
- 10:52 AM Revision 6932: schemas/Makefile: Installation: public: Generalized public%/publish to %/publish so that public schema versions don't have to start with public_
- 10:34 AM Revision 6931: schemas/Makefile: Installation: %/uninstall: Also display schema delete confirmation for schemas whose name is just the version suffix (r<revision #>)
- 10:32 AM Revision 6930: schemas/Makefile: Merged public%/uninstall and %/uninstall
- 09:49 AM Revision 6929: lib/common.Makefile: Added version target, which prints the current $(version) value
- 09:36 AM Revision 6928: schemas/Makefile: Installation: public: public%/uninstall: Fixed bug where need to remove the *specified* version of the public schema, not public itself. Generalized $(confirmRmPublicSchema) so it could also be used for named versions of the public schema. Inlined $(rmPublicSchema) since it's now only used in one place.
- 09:26 AM Revision 6927: lib/common.Makefile: Revisions: $(version): Use just the revision # to avoid cluttering the schema and log file names with long datetime strings
- 09:25 AM Revision 6926: schemas/Makefile: public%/install: schema comment: Include current date/time after version
- 09:20 AM Revision 6925: lib/common.Makefile: Replaced no longer used $(date) with function to generate human-readable text date (rather than date to put in filename). Removed leading zeros from date and hour. Added timezone.
- 09:07 AM Revision 6924: backups/Makefile: Removed no longer used $(dateFmt), $(mtime)
- 08:59 AM Revision 6923: backups/Makefile: Removed %.backup/rotate, because this incorrectly causes the current time rather than the version to be used in the backup filename. The version should instead be specified in the backup filename when it's created.
- 08:51 AM Revision 6922: schemas/Makefile: Installation: public: Added public%/publish to replace the current public schema with the given version
- 08:37 AM Revision 6921: schemas/Makefile: Installation: public: public/uninstall: Added public%/uninstall as a target to allow uninstalling versions of the public schema
- 08:30 AM Revision 6920: schemas/Makefile: Installation: public: public%/install: Add a comment on the schema containing the versioned schema name, so that if the schema is later renamed to just public (i.e. "published" as the current version), it will still be possible to tell which version the public schema came from
- 08:22 AM Revision 6919: schemas/Makefile: Installation: public: Added public%/install, to install a version of the public schema
- 07:59 AM Revision 6918: schemas/Makefile: Removed unused $(os)
- 07:58 AM Revision 6917: schemas/Makefile: Removed unused $(SED)
- 06:22 AM Revision 6916: Moved schemas-related commands from root Makefile to schemas/Makefile
- 06:15 AM Revision 6915: Makefiles: Factored out common vars/functions into lib/common.Makefile
- 05:59 AM Revision 6914: root Makefile: $(psqlNoSearchPath): Merged $(psqlAsBien) into it because it's the only place $(psqlAsBien) is used
- 05:56 AM Revision 6913: root Makefile: $(psqlAsBien): Use psql_script_vegbien instead of psql_vegbien, which adds $(psqlOpts) itself
- 05:50 AM Revision 6912: schemas/Makefile: Include lib/common.Makefile
- 05:23 AM Revision 6911: inputs/import.stats.xls: Reformatted so the first by column import and the comparison by row import will fit on the same page when printed on portrait-mode letter paper
- 05:10 AM Revision 6910: inputs/import.stats.xls: Changed import type labels to By row/By column so they would fit into one field, leaving the extra field free to contain the revision #
- 05:02 AM Revision 6909: lib/common.Makefile: Revisions: Allow $(version) to be overridden in the environment, so that the public schema and all log files share the same, pregenerated version
- 04:16 AM Revision 6908: schemas/vegbien.sql: Merged provider_view, provider_count, and owner_count into provider_count, using the combining query for Brad's data providers page at <http://bien.nceas.ucsb.edu/bien/people/data-providers/>
- 01:23 AM Revision 6907: schemas/vegbien.sql: sync_taxon_trait_to_view(): Changed pkey to index because there can be multiple values of the same taxon's trait from different observations
- 01:16 AM Revision 6906: mappings/Makefile: VegCore.csv: Filter out the VegCore tables so they are not matched as terms. This is necessary because some terms have the same name as a table, but the term should be the match rather than the table.
- 12:29 AM Revision 6905: sql.py: DbConn.col_info(): raising sql_gen.NoUnderlyingTableException: Fixed bug where also need to catch DoesNotExistException, which is thrown by ::regclass
- 12:26 AM Revision 6904: sql.py: DbConn.col_info(): Fixed bug where need to run run_query() recoverably, because this query throws an exception if the column's table does not exist (the information_schema query just returned no rows)
- 12:22 AM Revision 6903: sql.py: DbConn.col_info(): Fixed bug where need to use pg_get_expr() on pg_attrdef.adbin instead of shortcut field adsrc, because adsrc does not include schema qualifiers on table names (including strings passed to `nextval('..._seq'::regclass)`)
12/18/2012
- 11:42 PM Revision 6902: sql.py: DbConn.col_info(): Fixed bug where need to pass through cacheable param to run_query()
- 11:41 PM Revision 6901: sql.py: DbConn.col_info(): Fixed bug where need to use .to_str(self) instead of self.esc_value() because self.esc_value() expects a value, not a sql_gen.Literal instance
- 11:34 PM Revision 6900: sql.py: DbConn.col_info(): Fixed bug where self needs to be used everywhere that db normally is, because this is a DbConn method rather than a global function
- 11:31 PM Revision 6899: sql.py: DbConn.col_info(): For PostgreSQL, use pg_catalog tables directly instead of their views in information_schema. This allows using ::regclass to look up the table in the search_path, and fixes a bug in imports with an explicit public schema where column types were looked up in public instead of public.<version>. Also don't wrap default using sql_gen.as_Code() when it's None (indicating no default value, aka default=NULL), because this value is interpreted specially by sql_gen.TypedCol.
- 11:06 PM Revision 6898: inputs/Makefile: Input data: $(rsyncSrcs): Also include log files other than install.log.sql
- 09:41 PM Revision 6897: import_all: Run all imports (not just the main datasources' import) with $import_source turned off, so that the Source tables will not be imported a second time when the datasource's main tables are imported. Note that it's not necessary to wait for asynchronous commands after the jobs for the main import are started (so that $import_source is not unset until after they are started), because with_all does not return until all jobs are started and have noted the $import_source setting in effect in the shell environment.
- 09:32 PM Revision 6896: import_all: Source tables import: Fixed bug where need to use $all option to with_all to also include special datasources starting with "."
- 09:23 PM Revision 6895: make_analytical_db: Also create taxon_trait materialized view
12/17/2012
- 08:17 PM Revision 6894: inputs/*/*/map.csv: Reverted special OMIT mappings for input columns that have the same name as a VegCore table and have not yet been mapped to a VegCore term
- 08:06 PM Revision 6893: mappings/Makefile: VegCore.csv: Filter out the VegCore tables so they are not matched as terms. This is necessary because some terms have the same name as a table, but the term should be the match rather than the table.
- 08:04 PM Revision 6892: mappings/VegCore.csv: Changed line endings to \r\n to match the output of filter_out_ci
- 05:51 PM Revision 6891: inputs/CTFS/TaxonOccurrence/map.csv: Mapped SpeciesAuthority
- 04:59 PM Revision 6890: backups/Makefile: Synchronization: $(remote): Fixed bug where need trailing / at end of path
- 04:32 PM Revision 6889: backups/Makefile: Synchronization: $(remote): Updated path to backups
- 04:30 PM Revision 6888: README.TXT: Data import: On jupiter: Updated path to backups
- 04:25 PM Revision 6887: README.TXT: Installation: Added command to change to the directory of the checked out files
- 04:24 PM Revision 6886: README.TXT: Installation: Added command to check out files from svn
- 03:51 PM Revision 6885: schemas/vegbien.sql: Added taxon_trait materialized view
- 02:43 PM Revision 6884: mappings/Veg+-VegCore.csv: Sources: Removed redundant bien2_ prefix from bien2_staging subnamespace
- 02:21 PM Revision 6883: schemas/vegbien.sql: trait: trait_unique: Removed value and units because there should only be one value of a trait for each taxonoccurrence
- 02:18 PM Revision 6882: schemas/vegbien.sql: Reattached trait to taxonoccurrence instead of taxonlabel, because the TraitObservation traits data is actually associated with a particular occurrence (plant observation complete with location, date, etc.), rather than just a taxon
- 01:31 PM Revision 6881: Added inputs/bien2_traits/
- 01:29 PM Revision 6880: mappings/VegCore-VegBIEN.csv: Mapped traits-related DwC terms measurementType, measurementValue, measurementUnit
- 12:34 PM Revision 6879: schemas/vegbien.ERD.mwb: Added trait table to ERD
- 12:25 PM Revision 6878: schemas/vegbien.sql: trait: Added trait_unique unique index
- 12:19 PM Revision 6877: schemas/vegbien.sql: trait: Added units field
- 12:14 PM Revision 6876: schemas/vegbien.sql: trait: Renamed type to name because TraitObservation stores trait names rather than types
- 12:07 PM Revision 6875: schemas/vegbien.sql: trait: Linked to taxonlabel instead of stemobservation, because TraitObservation's traits are taxon-level and stem-level traits currently go in named fields instead of a stem traits table
- 11:45 AM Revision 6874: inputs/.TNRS/tnrs_*/map.csv: Remapped Source to OMIT so it won't match to the Source table
- 11:37 AM Revision 6873: inputs/.TNRS/tnrs_other/map.csv: Updated for new VegCore terms, which include Source as a table name. This field will need to be remapped so it doesn't collide with the table name.
- 10:04 AM Revision 6872: inputs/import.stats.xls: Updated import times
- 10:01 AM Revision 6871: README.TXT: Data import: Added step to check that the source table contains entries for all inputs
12/14/2012
- 01:01 PM Revision 6870: Regenerated vegbien.ERD exports
- 12:52 PM Revision 6869: make_analytical_db: Also populate owner_count
- 12:51 PM Revision 6868: make_analytical_db: Generate provider_count before analytical_aggregate because it's much faster
- 12:50 PM Revision 6867: schemas/vegbien.sql: Added materialized view owner_count, generated from owner_count_view
- 12:21 PM Revision 6866: make_analytical_db: Also populate provider_count
- 12:20 PM Revision 6865: schemas/vegbien.sql: Added materialized view provider_count, generated from provider_count_view
- 12:09 PM Revision 6864: schemas/vegbien.sql: Added provider_count_view for counts of occurrences per top-level provider
- 11:56 AM Revision 6863: Regenerated mappings/VegCore.htm
- 11:52 AM Revision 6862: Regenerated mappings/VegCore.htm
- 11:10 AM Revision 6861: Regenerated mappings/VegCore.htm
- 10:39 AM Revision 6860: schemas/vegbien.sql: provider_view: Sort NULL sourcetype last
- 10:36 AM Revision 6859: schemas/vegbien.sql: Added provider_view, which combines source and sourcename
- 10:31 AM Revision 6858: schemas/vegbien.sql: sourcename: Gave public_ SELECT permissions
- 10:17 AM Revision 6857: Regenerated mappings/VegCore.htm
- 10:15 AM Revision 6856: README.TXT: Maintenance: VegCore data dictionary: Regenerate everything in mappings/ that changes when VegCore.htm changes (such as VegCore.tables.redmine) instead of just VegCore.csv
- 09:29 AM Revision 6855: inputs/*/Source/map.csv without mappings: Added referenceType, etc. mappings. This also ensures that the source table entry for the datasource will be created before the herbaria list is imported, causing all top-level datasources to sort at the top of the source table.
- 09:02 AM Revision 6854: schemas/vegbien.sql: Granted the public_ user read-only access to the contents of the source table
- 08:53 AM Revision 6853: root Makefile: PostgreSQL: $(editPhppgadmin): Ignore errors if patch has already been applied
- 08:52 AM Revision 6852: lib/phpPgAdmin.config.inc.php.diff: Remove context so segment matching would depend only on the $conf['extra_login_security'] line itself
- 08:29 AM Revision 6851: mappings/Makefile: Added VegCore.tables.redmine, which contains the Redmine-formatted list of VegCore tables to paste into <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Tables>
- 08:18 AM Revision 6850: mappings/: Removed no longer used VegCore.redmine. VegCore.csv is now generated from the Redmine page instead of the other way around.
- 08:12 AM Revision 6849: mappings/Makefile: Added VegCore.tables.csv, which contains all the tables in the VegCore data dictionary
- 06:59 AM Revision 6848: README.TXT: Data import: backups/fix_perms: Run using sudo to also change permissions on files owned by the bien user, and to change the owner of files owned by you to the bien user
- 06:45 AM Revision 6847: Regenerated mappings/VegCore.csv, which adds categories
- 05:47 AM Revision 6846: README.TXT: Maintenance: Added instructions to regenerate mappings/VegCore.csv whenever the VegCore data dictionary page is changed
- 05:41 AM Revision 6845: mappings/Makefile: Generate VegCore.csv from the VegCore data dictionary page by extracting all HTML anchors (in Redmine, each section heading, and therefore each VegCore term, gets its own anchor)
- 05:34 AM Revision 6844: mappings/VegCore.csv: Changed line endings to \n to match what sed generates from the VegCore data dictionary page
- 05:31 AM Revision 6843: mappings/VegCore.csv: Removed informational columns, because this information is now maintained on the VegCore data dictionary page at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore>
- 05:11 AM Revision 6842: mappings/Veg+-VegCore.csv.csv: Removed hypothetical terms which are not in use by any VegBIEN datasource
- 05:00 AM Revision 6841: mappings/Veg+-VegCore.csv: habit: Remapped to growthForm, which replaces verbatimGrowthForm
- 04:59 AM Revision 6840: mappings/Veg+-VegCore.csv.csv: Removed hypothetical terms which are not in use by any VegBIEN datasource
- 04:50 AM Revision 6839: mappings/VegCore.csv: BIEN2 terms: Added sub-namespaces (bien_web, geoscrub, etc.) to source URLs
- 04:15 AM Revision 6838: dict2redmine: redmine_add_links(): Hyperlink just the source name, not also the () around it
- 03:54 AM Revision 6837: dict2redmine: RedmineDictWriter: Use h2 instead of h3 for the term name so that the term will be normal-sized instead of smaller in the Redmine table of contents
- 03:52 AM Revision 6836: dict2redmine: Renamed redmine_url() to redmine_link() because it generates links, not URLs
- 03:49 AM Revision 6835: dict2redmine: redmine_add_links(): Put citations in () instead of [] to avoid conflicting with the Redmine syntax for internal links ( [[...]] )
- 03:18 AM Revision 6834: mappings/VegCore.csv: Terms: Removed namespace prefixes (dcterms:), because VegCore terms are globally unique within VegCore and there should not be multiple versions of the same VegCore term with different namespaces. Provenance is instead indicated in the Sources column, which contains not just a namespace but a full URL to each source term.
- 03:00 AM Revision 6833: dict2redmine: Hyperlink each term to its anchor in the data dictionary, rather than to its first source, which is not necessarily the definitive definition of the term. This also allows clicking the term to get its permalink in the address bar, rather than having to click the small, light gray paragraph mark next to the term name that Redmine provides.
- 02:57 AM Revision 6832: dict2redmine: redmine_add_links(): Fixed bug where need to avoid matching internal links ( [[...]] ) as citations ( [...] )
- 02:46 AM Revision 6831: mappings/VegCore.csv: Term names: Changed special characters to _ because Redmine doesn't support special characters in HTML anchors (it removes everything except letters, numbers, _, and -)
- 02:42 AM Revision 6830: mappings/Makefile: .Veg+-VegCore.csv.last_cleanup: Also canon the output (VegCore) column to the VegCore.csv vocabulary. ? prefixes are not a problem because there are always at least two alternatives listed for these terms, so canon will not modify the output field.
- 01:49 AM Revision 6829: psql_script_vegbien: Run psql_vegbien with `nice -n +5` to prevent CPU-intensive operations from slowing down the shell/UI
- 01:46 AM Revision 6828: inputs/import.stats.xls: Updated import times
- 01:37 AM Revision 6827: Regenerated inputs/CVS/taxonObservation_/new_terms.csv. Note that it includes mappings to terms which are not in mappings/VegCore-VegBIEN.csv, which are prefixed with *.
- 01:34 AM Revision 6826: input.Makefile: Maps validation: %/new_terms.csv: Undid incorrect change of column to filter terms out of. This actually needs to be the input column, even though unmapped_terms.csv is generated from the output column, because it's possible to have a mapping to a term which is not in mappings/VegCore-VegBIEN.csv, and such a term would show up in unmapped_terms.csv but should not be filtered out of new_terms.csv.
- 01:17 AM Revision 6825: lib/phpPgAdmin.login.php.diff: public_ user's password message: Print as its own message instead of appending it to $msg. Print it before any error message so it always appears at the top of the page.
- 12:51 AM Revision 6824: root Makefile: PostgreSQL: phpPgAdmin: Edit config file to allow passwordless logins. Edit login page to fill in public_ as the default username and add a message to leave the password blank for that user.
Also available in: Atom