Activity
From 01/11/2012 to 02/09/2012
02/09/2012
- 07:08 PM Revision 901: mappings/VegX-VegBIEN.organisms.csv: Fixed TaxonomicRankBelowSubspeciesEnum mapping to use plantname instead of placing the name directly inside plantconcept. This bug was probably left over from the switch from linking directly to plantname rather than to a plantconcept which has a plantname.
- 06:54 PM Revision 900: Renamed get_errors to error_stats and added counts of each error message
- 06:27 PM Task #352: create way to represent methods hierarchically in schema
- I added a basic plotmethod table. What hierarchical information do we want to add?
- 06:26 PM Task #350: implement methods in VegBIEN
- I added a basic plotmethod table
- 06:24 PM Revision 899: Regenerated vegbien.ERD exports
- 06:22 PM Revision 898: vegbien.sql: Added plotmethod table
- 05:42 PM Task #348: 1st draft of schema
- e-mail from Martha Narro on 2012-2-9:
* 1:1 constraint on taxonOccurrence has been dropped, so that topic is final... - 05:40 PM Task #345: integrate GNRS into VegBIEN
- See "iPlant Geospatial Roadmap":https://pods.iplantcollaborative.org/wiki/display/SOT/Geospatial+Roadmap+%28revised%2...
- 05:36 PM Task #346: add georeferencing support to schema
- "DwC georeferencing fields":http://code.google.com/p/darwincore/wiki/Location#Georeference_Terms:
* decimalLatitude
... - 05:34 PM Task #296 (Resolved): Direct mapping from native salvias_plots MySQL database to VegBIEN
- SALVIAS stems have been mapped. Mappings are in SVN under "@inputs/SALVIAS/maps@":https://projects.nceas.ucsb.edu/nce...
- 05:31 PM Task #304: Complete full dataset imports to VegBIEN via VegX of NYBG and SALVIAS
- First full NYBG load is complete, with 0.12% error rate. Full stats under [[Benchmarking]].
- 05:30 PM Task #313 (Resolved): Import full NYBG data
- First full NYBG load is complete, with 0.12% error rate. Full stats under [[Benchmarking]].
- 05:29 PM Task #323 (Resolved): map NYBG data
- First full NYBG load is complete, with 0.12% error rate. Full stats under [[Benchmarking]].
- 05:28 PM Task #342 (Resolved): make DwC_BIEN schema
- [[DwC schema]] has a list of fields in DwC-BIEN, mappings from different versions of DwC, and mappings to VegBIEN
02/07/2012
- 05:22 PM Revision 897: input.Makefile: Renamed $(test) variable (containing the command to run a test) so it wouldn't conflict with test environment variable
- 05:10 PM Revision 896: mappings to VegBIEN: Don't map notes to taxondetermination because some organisms have notes but no species name
- 05:08 PM Revision 895: mappings/Makefile: Fixed bug in rules for cleaning up core maps whenever they change, which had the target and prerequisite reversed
- 04:39 PM Revision 894: input.Makefile: nolog option defaults to on when test is on
- 04:36 PM Revision 893: input.Makefile: Fixed bug where no log file was being created, even when nolog was off
- 04:11 PM Revision 892: Replaced all type(...) == str with util.is_str(...) to properly treat Unicode objects as strings
- 04:08 PM Revision 891: xml_dom.py: minidom.Element.write_opening(): Use new Attr.__str__() method for printing attributes
- 04:06 PM Revision 890: bin/map: For XML inputs, use xpath.get()'s new multiple result support to iterate over elements matching the root, rather than just iterating over the first matching element's siblings. This fixes the broken 2-step tests for stems, which didn't filter by the root's attributes.
- 04:04 PM Revision 889: xpath.py: get(): Fixed bug where limit was not passed to recursive get() calls
- 04:02 PM Revision 888: xml_dom.py: by_tag_name(): Iterate forwards over children unless last_only optimization turned on. Added Attr.__str__() and __repr__() for debug-printing Attrs.
- 03:10 PM Revision 887: xpath.py: get(): Turn off last_only optimization when limit > 1
- 03:07 PM Revision 886: xpath.py: get(): Added full support for returning multiple matches
- 02:33 PM Revision 885: xpath.py: get(): Added basic structure for returning multiple matches. Added limit parameter to select one or many matches.
- 02:13 PM Revision 884: input.Makefile: test/VegBIEN.%.2-step.xml: Use the core map specific to the test's table instead of the main core map
- 02:10 PM Revision 883: bin/map: Print error if map root not found inXML input
- 01:52 PM Revision 882: mappings/: Removed mappings used by old tester
- 01:48 PM Revision 881: inputs/SALVIAS/maps/VegX.*.csv: Replaced symlinks with actual files
- 01:45 PM Revision 880: Removed old tester
- 01:44 PM Revision 879: Switched to using inputs/test as main test target
- 01:38 PM Revision 878: Added SALVIAS-CSV tests
- 01:31 PM Revision 877: Added NYBG-CSV tests
- 01:28 PM Revision 876: input.Makefile: Run separate tests for each map spreadsheet (input table) rather than all tables at once. This will make it possible to test CSV inputs, which have one CSV per table.
- 12:54 PM Revision 875: Added NYBG-CSV input
- 12:47 PM Revision 874: inputs/Makefile: Fixed forwarding of empty targets to subdirs
- 12:20 PM Revision 873: Regenerated vegbien.ERD exports
- 12:17 PM Revision 872: vegbien.sql: Added morphospecies table
- 11:53 AM Revision 871: vegbien.ERD.mwb: Fixed lines
- 11:45 AM Revision 870: vegbien.sql: Removed the taxonOccurrence:aggregateOccurrence 1:1 constraint
02/06/2012
- 05:22 PM Revision 869: sql.py: truncate(): Use run_raw_query() instead of run_query() because truncate() does not use the recover functionality of run_query(). Also, in the profiling output, this separates the "normal" SQL statements (which use run_query()) from the "core" SQL statements (which use run_raw_query()).
- 05:09 PM Revision 868: vegbien.sql: Added indexes for each field in party used in duplicate elimination (for use by sql.put()'s DuplicateKeyException handler)
- 04:58 PM Revision 867: sql.py: run_raw_query(): In debug mode, print query after params have been substituted in
- 04:55 PM Revision 866: sql.py: Fixed index_cols() to handle UNIQUE indexes with expressions, whose column names are stored in a different format
- 04:04 PM Revision 865: sql.py: Print warning if SELECT statement missing a WHERE, LIMIT, or OFFSET clause. Changed bin/map DB input get-all-rows statement to pass start=0 to suppress this warning for that statement.
- 03:51 PM Revision 864: db_xml.py: Added start option to get() that passes through to sql.select()
- 03:50 PM Revision 863: sql.py: Added start option to select() to set the OFFSET
- 03:27 PM Revision 862: sql.py: If run_raw_query.debug flag is set, print each query executed (on a single line)
- 03:25 PM Revision 861: strings.py: Added one_line() function to make a string all on one line
- 03:20 PM Revision 860: strings.py: Renamed one_line() to remove_extra_newl() to better reflect what it does
- 03:19 PM Revision 859: bin/map: Don't print Done after an action in debug logging mode because it messes up newlines when more debugging info is printed right after it
- 02:45 PM Revision 858: input.Makefile: Added nolog option to disable creating a log file, e.g. for debugging runs
- 01:57 PM Revision 857: xml_dom.py: Remove extra newlines from single-line strings (bin/map doesn't need to do this itself anymore)
- 01:56 PM Revision 856: strings.py: Added is_multiline() and one_line() for removing extra newlines from single-line strings
- 01:48 PM Revision 855: bin/map: In debug mode, print input XPath's XML tree all on one line
- 01:22 PM Revision 854: sql.py: Switched try_insert() to use index_cols() instead of constraint_cols() for "duplicate key value violates unique constraint" errors because they can also be generated by UNIQUE indexes (and there is a UNIQUE index for every UNIQUE constraint)
- 01:14 PM Revision 853: sql.py: Added index_cols() to get cols used by an index (similar to constraint_cols())
- 01:00 PM Revision 852: vegbien.sql: Fixed duplicate elimination for party to use a UNIQUE index with COALESCE for nullable fields
- 01:00 PM Revision 851: sql.py: Fixed bug in try_insert() where DuplicateKeyException was passed only cols[0] instead of cols array
- 12:49 PM Revision 850: vegbien.ERD.mwb: Fixed lines
- 12:30 PM Revision 849: Added get_errors to select just the error messages from `map` output
- 11:59 AM Revision 848: Added profile_stats to analyze a profiling statistics file
- 11:59 AM Revision 847: bin/map: Added profile_to option which turns on profiling to the specified file
- 11:13 AM Revision 846: bin/map: Added "if __name__ == '__main__': main()" idiom so file can be included as well as run. This will be useful for profiling.
- 11:12 AM Revision 845: dates.py: Fixed strftime() to pad years and days with leading zeros as datetime.strftime() does
- 11:01 AM Revision 844: dates.py: Work around strftime() bug that can't deal with 2/29 on a leap year
- 10:44 AM Revision 843: xml_func.py: Added FormatException for SyntaxExceptions generated by strftime() (which are often Python bugs)
- 10:25 AM Task #317: Direct mapping from VegX to VegBIEN
- Mappings have been made for SALVIAS VegX fields
- 10:23 AM Task #333: add traits table to VegBIEN
- VegBank had an existing trait table, which was not included in the ERD. We can modify that table for our needs.
- 10:23 AM Task #332: switch to VegX 1.5.3
- VegX 1.5.3 is available, so we're switching to that instead
- 10:22 AM Task #313: Import full NYBG data
- Almost done with the first full load of the NYBG data. Currently takes 2 days.
- 09:46 AM Revision 842: Added schemas/vegbank.revised.sql. Initial version has all "character varying" types replaced with text.
- 09:45 AM Revision 841: vegbien.sql: Replaced all "character varying" types with text, removing the length limits. Note that in PostgreSQL, text and "character varying" are stored the same way internally, so this does not affect performance or indexes.
02/04/2012
- 05:28 PM Revision 840: xml_dom.py: Added documentation labels to each section
- 05:21 PM Revision 839: xml_dom.py: Fixed bug in NodeTextEntryIter where an entry containing an element instead of a text node would be returned as the whole entry, instead of the value *of* the entry
- 04:56 PM Revision 838: bin/map: Added support for starting import at a specific row. Refactored row-processing code with and without a map to use a common process_rows() function (with the previous process_rows() being renamed to map_rows()).
- 04:14 PM Revision 837: bin/map: Use new util.cast()
- 04:14 PM Revision 836: util.py: Added cast() to cast a value while passing None through
- 04:09 PM Revision 835: bin/map: Print row # of rows with errors
- 03:27 PM Task #349: make ERD of full schema
- Full ERD is available at [[VegBIEN schema]]. A few tables are not yet included, because their lines to existing table...
- 03:25 PM Task #354 (Resolved): list of deliverables for next week (2/6-2/10)
- See [[2012-02-03 conference call]]
- 03:22 PM Revision 834: sql.py: Fixed error in pkey() where recover was not passed as a named parameter to run_query()
- 03:12 PM Revision 833: sql.py: Added documentation labels to each section
- 03:10 PM Revision 832: sql.py: Added documentation labels to each section
- 03:03 PM Revision 831: db_xml.py: Used new sql.py recover functionality
- 03:03 PM Revision 830: sql.py: Added ability to recover from database errors so you don't get the error "InternalError: current transaction is aborted, commands ignored until end of transaction block"
- 02:25 PM Revision 829: vegbien.sql: Removed taxonoccurrence.taxoninferencearea because it's duplicated in aggregateoccurrence.inferencearea
- 02:18 PM Revision 828: bin/map: Highlight the "input row" and "output row:" labels in error messages
- 02:17 PM Revision 827: xml_func.py: Highlight nodes that were commented out because of errors
- 02:17 PM Revision 826: exc.py: Print exceptions with the first line highlighted in red
- 02:16 PM Revision 825: term.py: Added emph() and error()
- 01:51 PM Revision 824: vegbien.ERD.mwb: Fixed lines
- 01:47 PM Revision 823: vegbien.ERD.mwb: Fixed lines
- 01:46 PM Revision 822: vegbien.ERD.mwb: Added note, notelink, and revision
- 01:43 PM Revision 821: vegbien.ERD.mwb: Added embargo to diagram
- 01:27 PM Revision 820: Regenerated vegbien.ERD exports
- 01:24 PM Revision 819: vegbien.ERD.mwb: Fixed lines. Added "Core subset" and "Other tables" labels.
- 01:03 PM Revision 818: xml_func.py: Changed _date func to use new dates.strftime(), which can handle years before 1900
- 01:03 PM Revision 817: Added dates.py to handle date/time manipulation, such as fixing Python's broken strftime() that can't handle years before 1900
- 01:02 PM Revision 816: Regenerated mappings/for_review/DwC-VegBIEN.specimens.csv
02/03/2012
- 06:15 PM Revision 815: Regenerated vegbien.ERD exports
- 06:12 PM Revision 814: vegbien.ERD.mwb: Added reference and party tables
- 05:49 PM Revision 813: vegbien.ERD.mwb: Fixed lines
- 05:48 PM Revision 812: filter_ERD.csv: Remove fkeys to heavily-linked tables (reference, party)
- 05:33 PM Task #347 (Resolved): revise milestones
- Brad and Martha have approved it
- 03:15 PM Task #347: revise milestones
- My revisions are on the wiki under *[[Milestones]]*
- 01:57 PM Task #347 (Resolved): revise milestones
- 05:30 PM Revision 811: Added to_do/milestones.doc
- 05:27 PM Revision 810: Renamed milestones.doc to timeline.doc
- 05:24 PM Revision 809: Added schemas/filter_ERD.csv and use it when generating vegbien.my.sql
- 05:16 PM Revision 808: vegbien.ERD.mwb: Added cover* to main diagram
- 04:53 PM Revision 807: vegbien.ERD.mwb: Started adding additional tables "below the fold" on the 2nd page
- 04:34 PM Revision 806: vegbien.ERD.mwb: Fixed lines
- 04:26 PM Revision 805: vegbien.ERD.mwb: Moved legend to top left to make room for more misc tables. Organized legend by location on diagram.
- 04:14 PM Revision 804: vegbien.ERD.mwb: Added soilobs table
- 04:07 PM Revision 803: vegbien.ERD.mwb: Added userdefined tables. Fixed lines.
- 03:57 PM Revision 802: vegbien.ERD.mwb: Changed location color to match VegBank ERD
- 03:53 PM Revision 801: vegbien.ERD.mwb: Added trait to diagram
- 03:38 PM Revision 800: vegbien.ERD.mwb: Added plantstatus to diagram. Added margins around diagram.
- 03:18 PM Task #320: Convert user-defined VegX fields to first-class fields
- Brad's meeting notes on 2012-2-3:
* Aaron to make whatever modifications are needed to match VegBien schema
* Sub... - 02:49 PM Revision 799: Added milestones.doc
- 02:04 PM Task #346: add georeferencing support to schema
- * support DwC georeferencing fields
- 01:57 PM Task #346 (New): add georeferencing support to schema
- 02:03 PM Task #354 (Resolved): list of deliverables for next week (2/6-2/10)
- 02:02 PM Task #353 (Resolved): add terms from previous versions of DwC to DwC-BIEN
- 02:02 PM Task #352 (Resolved): create way to represent methods hierarchically in schema
- 02:01 PM Task #351 (Resolved): list of fields and method attributes needed to know whether can combine data from different plots
- * critical methodological elements
- 02:00 PM Task #350 (Resolved): implement methods in VegBIEN
- * middle ground between descriptions and ontological metadata
- 01:59 PM Task #349 (New): make ERD of full schema
- 01:58 PM Task #348 (Resolved): 1st draft of schema
- * plots and specimens data
- 01:56 PM Task #345 (Resolved): integrate GNRS into VegBIEN
- * geovalidation in scope
* georeferencing out of scope unless existing implementation
- 01:56 PM Task #344 (Resolved): contact John about GNRS/georeferencing
- * Using PostGIS/R?
- 01:55 PM Task #343 (Resolved): integrate TNRS into VegBIEN
- 12:39 PM Revision 798: DwC mappings: Fixed syntax of _date XML funcs to not wrap dates twice in a _date func
- 12:38 PM Revision 797: xml_func.py: Fixed bug in SyntaxException constructor where the cause was not passed to ExceptionWithCause
- 12:24 PM Revision 796: xml_dom.py: Override Node.__repr__ and Element.__repr__ to make sure self.toprettyxml() is used in all cases where a Node is converted to a string
- 11:57 AM Revision 795: util.py: Don't print type name in ConstraintError message because sometimes an entry tuple (e.g. from a dict) is checked, and then the type isn't useful
- 11:53 AM Revision 794: xml_func.py: Use conv_items(str, ...) in XML funcs that require strings, but don't otherwise use conv_items()
- 11:50 AM Revision 793: xml_func.py: Use xml_dom.TextEntryOnlyIter to check inputs for XML funcs that only handle strings, not whole elements
- 11:49 AM Revision 792: xml_dom.py: Added TextEntryOnlyIter to check that values returned by NodeTextEntryIter are strings from text nodes
- 11:48 AM Revision 791: util.py: Added is_str() and CheckedIter
01/31/2012
- 07:52 PM Revision 790: input.Makefile: Log each import to a new log file named according to the current time
- 07:38 PM Revision 789: xml_func.py: SyntaxException: Use exc.ExceptionWithCause
- 07:37 PM Revision 788: exc.py: Added str_() to convert an Exception to a string. Use str_() in print_ex().
- 07:31 PM Revision 787: xml_func.py: conv_items(): Only trap type conversion errors, not errors in map_items(). A previous bug incorrectly interpreted an error caught by map_items() as a SyntaxException.
- 07:23 PM Revision 786: xml_func.py: _date func: Fixed bug where setting defaults of a date part and removing zeros needed to happen after elements were converted to int. Refactored to use conv_items().
- 06:58 PM Revision 785: Makefiles: Added documentation labels to each section
- 06:55 PM Revision 784: Makefile: Added documentation labels to each section
- 06:49 PM Revision 783: input.Makefile: Added documentation labels to each section
- 06:44 PM Revision 782: xml_dom.py: Fixed conversion of minidom.Node to a string to use toprettyxml() again
- 06:40 PM Revision 781: Added inputs/SALVIAS-CSV/maps/VegBIEN.* maps
- 06:39 PM Revision 780: input.Makefile: If a test doesn't yet have accepted output, print its current output
- 06:24 PM Revision 779: mappings/Makefile: Only clean up a core map (intermediate format->VegBIEN) when it's modified. Note that the .last_cleanup files need to be under version control to prevent cleanups upon running any map command in a new checkout.
- 06:07 PM Revision 778: xml_func.py: Support dates with year/month/day == 0
- 06:06 PM Revision 777: input.Makefile: Print informative message if test failure is ignored
- 05:47 PM Revision 776: input.Makefile: Print path to test output so you can tell which directory a test is running in
- 05:45 PM Revision 775: input.Makefile: Fixed 2-step tests, which were still using $(root)/map when all bin/map calls were supposed to be routed through $(map)
- 05:05 PM Revision 774: xml_dom.py: Print simple XML elements on one line
- 04:33 PM Revision 773: Added inputs/SALVIAS-CSV/maps
- 04:33 PM Revision 772: input.Makefile: Added import support for CSV and XML inputs (no test support yet)
- 03:48 PM Revision 771: inputs/SALVIAS-CSV: Added src files
- 03:47 PM Revision 770: inputs/SALVIAS-CSV: Added src files
- 03:42 PM Revision 769: inputs: Removed no longer needed svn:ignore
- 03:41 PM Revision 768: input.Makefile: Coded svn:ignore directly in Makefile
- 03:40 PM Revision 767: inputs/Makefile: Added %-add target to add new inputs
- 03:35 PM Revision 766: input.Makefile: Added svn_props rule for updating svn:ignore of dir and subdirs
- 02:42 PM Revision 765: vegbien.sql: Made aggregateoccurrence:taxonoccurrence relationship 1:1
- 02:03 PM Revision 764: Added inputs/CTFS/src, with subset of files that will be in version control
- 02:01 PM Revision 763: mappings/for_review/DwC-VegBIEN.specimens.csv: Updated
01/30/2012
- 07:14 PM Revision 762: inputs/NYBG: Map via DwC instead of VegX
- 07:12 PM Revision 761: mappings/DwC-VegBIEN.specimens.csv: Provide schema URLs for namespaces (in the column header)
- 07:11 PM Revision 760: input.Makefile: Deal with core maps (the main *-VegBIEN map) whose roots are other than organisms
- 07:09 PM Revision 759: bin/map: In the xml_func.process() for XML outputs, pass an on_error that outputs to ex_tracker.track()
- 07:08 PM Revision 758: xpath.py: set_id(): Deal with XPaths with an
- 07:07 PM Revision 757: xml_dom.py: replace_with_text(): Accept Node inputs for cases where e.g. an XML func with an error has been replaced with a comment
- 07:06 PM Revision 756: xml_func.py: _date: Year defaults to 1900
- 05:18 PM Revision 755: DwC-VegBIEN.specimens.csv: Output DwC as XML instead of CSV
- 05:05 PM Revision 754: xpath.py: Allow XPath names to be quoted literals, just like values. Handle : embedded in a name (even without quotes), often indicating a namespace.
- 04:55 PM Revision 753: Removed no longer needed lib/ERD-wiki.csv
- 04:54 PM Revision 752: Parser.py: In syntax errors, only print <END> at end of string if error was *at* end of string
- 04:50 PM Revision 751: bin/map: Print error message rather than assertion failure for CSV output, which isn't supported yet
- 04:47 PM Revision 750: input.Makefile: Don't abort tester if via test fails, because CSV output is not supported yet
- 04:15 PM Task #323: map NYBG data
- Infrastructure exists to do full NYBG loads, but there are many errors in the import
- 04:12 PM Task #338 (Resolved): put together samples of SALVIAS data that aggregates individuals and stems in different ways
- See [[SALVIAS issues]]
- 04:02 PM Revision 749: input.Makefile: Deal properly with more than one format that's being mapped via (e.g. DwC and VegX)
- 03:49 PM Revision 748: Renamed mappings/DwC-VegBIEN.organisms.csv back to mappings/DwC-VegBIEN.specimens.csv because it now uses /specimenreplicate as the output root
- 03:43 PM Revision 747: Renamed mappings/DwC-VegBIEN.organisms.csv back to mappings/DwC-VegBIEN.specimens.csv because it now uses /specimenreplicate as the output root
- 03:41 PM Revision 746: mappings/DwC-VegBIEN.organisms.csv: Fixed syntax error in a mapping
- 03:35 PM Revision 745: mappings/DwC-VegBIEN.organisms.csv: Sorted
- 03:34 PM Revision 744: mappings/Makefile: Sort DwC-VegBIEN.organisms.csv
- 03:33 PM Revision 743: inputs/NYBG/maps/VegBIEN.organisms.csv: Regenerated
- 03:12 PM Revision 742: mappings/DwC-VegBIEN.organisms.csv: Sort on output column
- 03:04 PM Revision 741: Renamed mappings/DwC-VegBIEN.specimens.csv to DwC-VegBIEN.organisms.csv because it (currently) uses /taxonoccurrence as the output root
- 03:03 PM Revision 740: Renamed mappings/DwC-VegBIEN.specimens.csv to DwC-VegBIEN.organisms.csv because it (currently) uses /taxonoccurrence as the output root
- 03:01 PM Revision 739: Makefile: Support recursing into subdir when no target is specified (e.g. make mappings/)
- 02:55 PM Revision 738: union: Merge rows (and column labels) using maps.merge_mappings()
- 02:51 PM Revision 737: join: Merge the column labels as well
- 02:51 PM Revision 736: maps.py: Eliminate duplicates when merging values in the same column
- 02:06 PM Revision 735: join: Moved mappings-specific merge functionality into maps.merge_mappings()
- 01:55 PM Revision 734: join: Use merge_rows() from new maps.py
- 01:54 PM Revision 733: Added new library maps.py for map spreadsheet manipulation
- 01:38 PM Revision 732: join: Merge comments of input map and join map
- 01:37 PM Revision 731: join: Report which input mappings are missing a mapping in the join map
- 01:03 PM Revision 730: join: Report which input mappings are missing a mapping in the join map
- 12:28 PM Revision 729: inputs/NYBG/maps/VegX.organisms.csv: Added note that the primary key has NULL values in some rows
- 12:24 PM Revision 728: input.Makefile: Preserve as many intermediate files as possible (make likes to delete intermediates if they aren't marked as .PRECIOUS)
- 11:46 AM Revision 727: sort: Sort empty strings last so that inputs with no mapping go at the end of the map spreadsheet
- 11:24 AM Revision 726: VegBIEN-VegBank.csv: Updated for recent table renames
- 11:21 AM Revision 725: join: For input mappings with no match in the join map, include them in the output map with an empty mapping
- 11:05 AM Revision 724: input.Makefile: Generalized to handle mapping via any format, not just VegX
- 10:44 AM Revision 723: input.Makefile: Don't print message to accept output for failing 2-step tests, because they use another test's accepted output
01/28/2012
- 06:02 PM Revision 722: input.Makefile: Don't abort tester if only 2-step test fails, as it's often finicky
- 05:55 PM Revision 721: xml_func.py: Raise xml_func.SyntaxException for ValueErrors generated by date.strftime() (e.g. year out of range due to poor Y2K support in some OS implementations of strftime)
- 05:51 PM Revision 720: xml_func.py: Raise xml_func.SyntaxException for ValueErrors generated by datetime.date() (e.g. month out of range)
- 05:46 PM Revision 719: vegbien.sql: Added project.reference_id to namespace project names by datasource
- 05:39 PM Revision 718: input.Makefile: Import all tables at once by default
- 05:11 PM Revision 717: bin/map: Print "Inserted ... new rows into database" message to stdout rather than stderr so it can be stored in the test case output as a validation check
- 05:08 PM Revision 716: Accepted initial test output for NYBG/test/import.out.ref
- 05:06 PM Revision 715: bin/map: Clean up datasource input values
- 05:06 PM Revision 714: strings.py: Added std_newl() to convert line endings and cleanup() to process strings with extra or nonstandard whitespace
- 04:46 PM Revision 713: PostgreSQL-MySQL.csv: Deal with custom types
- 04:36 PM Revision 712: vegbien.sql: Added aggregateoccurrence.occurrencestatus_dwc field
- 04:22 PM Revision 711: Regenerated vegbien.ERD exports
- 04:21 PM Revision 710: vegbien.ERD.mwb: Added commclass table to ERD
- 04:11 PM Revision 709: vegbien.sql: Removed direct pointer from location to namedplace because locationplace already has this relationship and we don't want to have an extra pointer just for duplicate elimination
- 03:56 PM Revision 708: vegbien.ERD.mwb: Added stratummethod to ERD
- 03:53 PM Revision 707: vegbien.sql: Removed locationevent.stratummethod_id because the stratummethod is a per-stratum (or technically, per-stratumtype) field
- 03:28 PM Revision 706: PostgreSQL-MySQL.csv: Remove CHECK constraints
- 03:20 PM Revision 705: PostgreSQL-MySQL.csv: Remove functions and triggers
- 03:13 PM Revision 704: vegbien.sql: Ensure that aggregateoccurrence.count == 1 when the aggregateoccurrence has a plantobservation. Use a trigger to do this automatically.
- 02:51 PM Revision 703: README.TXT: Added command for reimporting data
- 02:48 PM Revision 702: README.TXT: Added instructions to sync ERD with vegbien.sql schema. Organized commands into categories.
- 02:23 PM Revision 701: Regenerated vegbien.ERD exports
- 02:22 PM Revision 700: Added BIEN_logo.png
- 02:19 PM Revision 699: vegbien.ERD.mwb: Added color group legend
- 01:59 PM Revision 698: Regenerated vegbien.ERD exports
- 01:58 PM Revision 697: vegbien.ERD.mwb: Fixed lines
- 01:57 PM Revision 696: Regenerated vegbien.ERD exports
- 01:57 PM Revision 695: vegbien.ERD.mwb: Fixed lines and moved plant to its own color category
- 01:52 PM Revision 694: vegbien.ERD.mwb: Added colors to ERD
- 01:34 PM Revision 693: vegbien.ERD.mwb: Simplified diagram by removing column types
- 12:56 PM Revision 692: schemas/Makefile: Don't generate for_ERD DDLs because the ERD is now synced with the full schema
- 12:54 PM Revision 691: vegbien.ERD.mwb: Synced with whole schema
- 12:54 PM Revision 690: vegbien.sql: Reordered fields in tables truncated in the ERD so that all removed fields are at the end of the table
- 12:44 PM Revision 689: schemas/Makefile: Generate MySQL version of vegbien.sql as well as vegbien.for_ERD.sql for eventual use in syncing the ERD with the whole schema
- 12:43 PM Revision 688: PostgreSQL-MySQL.csv: Added translations for syntaxes used by pg_dump
- 12:41 PM Revision 687: PostgreSQL-MySQL.csv: Added translations for syntaxes used by pg_dump
- 12:40 PM Revision 686: repl: All regexps are by default in multiline and ignore case mode
01/27/2012
- 04:59 PM Revision 685: Regenerated vegbien.ERD exports
- 04:59 PM Revision 684: vegbien.sql: Made planttag a child of plantobservation instead of plant, since tags change over time
- 04:39 PM Revision 683: Regenerated vegbien.ERD exports
- 04:37 PM Revision 682: vegbien.sql: Removed no longer used plantobservation.aggregateoccurrence_id
- 04:35 PM Revision 681: VegX-VegBIEN mapping: Link aggregateoccurrence to plantobservation via forward pointer rather than backward child-to-parent pointer
- 04:32 PM Revision 680: vegbien.sql: Made plantobservation.aggregateoccurrence_id optional because link will soon be going in the other direction
- 04:25 PM Revision 679: vegbien.sql: Removed taxonbinmethod table since its fields are now in aggregateoccurrence
- 04:14 PM Revision 678: vegbien.sql: Added taxonbinmethod fields to aggregateoccurrence
- 04:10 PM Revision 677: vegbien.sql: Added back aggregateoccurrence.stratum_id
- 04:05 PM Revision 676: vegbien.sql: Added stratum.area
- 04:03 PM Revision 675: vegbien.sql: Removed denormalized duplicate fields from stratum
- 03:58 PM Revision 674: vegbien.sql: Added plant and planttag tables
- 03:43 PM Revision 673: VegBIEN: Renamed stem to stemobservation
- 03:35 PM Revision 672: vegbien.sql: Removed specimenreplicate:taxonoccurrence 1:1 requirement
- 03:32 PM Revision 671: VegBIEN: Renamed individualplant to plantobservation
- 03:22 PM Revision 670: vegbien.sql: Updated table comments for specimenreplicate and specimen
- 03:19 PM Revision 669: vegbien.sql: Added specimen table to tie specimenreplicates together
- 03:12 PM Revision 668: VegBIEN: Renamed specimen to specimenreplicate
- 03:06 PM Revision 667: Remerged ERD DDL into ERD
- 03:04 PM Task #342 (Resolved): make DwC_BIEN schema
- 03:03 PM Task #341 (New): create verbal descriptions of edge cases
- test cases for whether model can appropriately accommodate data
- 02:55 PM Revision 666: Redoing commit that linked aggregateoccurrence forward to individualplant, allowing many taxonoccurrences (e.g. one for each specimen) to point to the same plant (e.g. that those specimens came from)
- 12:57 PM Revision 665: Added NYBG input
- 12:48 PM Revision 664: input.Makefile: Run tests with verbose output
- 12:48 PM Revision 663: bin/map: Fixed bug where verbose/debug flags were ignored and message were always printed.
- 12:39 PM Revision 662: bin/map: Added verbose and debug options. Added initial debug info.
- 12:39 PM Revision 661: xml_dom.py: Added is_simple() to determine whether every child recursively has no more than one child. Used is_simple() to print condensed XML when simple nodes are converted to a string.
- 11:58 AM Revision 660: vegbien.sql: Enforce 1:1 relationship between aggregateoccurrence<->individualplant and taxonoccurrence<->specimen
- 11:44 AM Revision 659: vegbien.sql: Changed individualplant UNIQUE constraint to enforce 1:1 relationship between aggregateoccurrence and individualplant
- 11:35 AM Revision 658: Undoing previous commit since it would prevent a plant from being tied to a data source, because the aggregateoccurrence pointer goes in the wrong direction
- 11:18 AM Revision 657: vegbien.sql: Added aggregateoccurrence.individualplant_id to make a 1:1 relationship between aggregateoccurrence and individualplant
- 01:55 AM Revision 656: input.Makefile: Generate VegBIEN.2-step.xml correctly from VegX.xml, by removing DB config env vars passed to map for that test case. Note that this causes the VegBIEN.2-step.xml test to fail, because the 2-step mapping does not yet match the 1-step mapping.
- 01:28 AM Revision 655: input.Makefile: Don't need to filter test output since stderr now goes to the screen
- 01:21 AM Revision 654: input.Makefile: Don't save *.err outputs for each test because this information is printed to the screen
01/26/2012
- 06:50 PM Revision 653: input.Makefile: Send echoed diff command to stdout of the make process (set -x echoes it to stderr)
- 06:48 PM Revision 652: input.Makefile: Write test stderr to .err file instead of test output, and tee it to stdout of the make process
- 06:29 PM Task #340 (Resolved): put together definitions of abstract BIEN terms
- * specimen
* voucher
* taxonoccurrence - 06:08 PM Revision 651: vegbien.sql: Updated name of UNIQUE constraint for specimen collectionnumber. Regenerated vegbien.ERD exports.
- 05:49 PM Revision 650: input.Makefile: Don't print "accept test" message when user aborted a test with Ctrl+C
- 05:34 PM Revision 649: inputs/SALVIAS/test: Accepted test outputs
- 05:33 PM Revision 648: input.Makefile: Also print message for accepting test output when diff fails
- 05:23 PM Revision 647: bin/map: Print a message when a database is successfully connected to
- 05:20 PM Revision 646: bin/map: Print a message when a database is successfully connected to
- 05:03 PM Revision 645: sql.py: Don't enclose PostgreSQL names in quotes because this disables case-insensitivity
- 04:58 PM Revision 644: sql.py: Use esc_name() to escape fields in SELECT statements
- 04:54 PM Revision 643: sql.py: Added esc_name() to escape identifiers like column names
- 04:27 PM Revision 642: vegbien.sql: Added comments to specimen.collectioncode_dwc and collectionnumber to differentiate them
- 04:22 PM Revision 641: vegbien.sql: Renamed authorspecimencode to collectionnumber to match its name in source data
- 04:18 PM Revision 640: input.Makefile: Use pipefail to cause a test to fail even when the output is filtered by grep. Print message for failing tests with command to run to accept the new test output.
01/24/2012
- 06:32 PM Task #339 (New): have "schema showdown"
- Discuss different meanings everyone has for various ecological terms, especially:
* taxonoccurrence
* specimen - 06:10 PM Revision 639: VegX-VegBIEN mapping: Map additional taxondetermination.determinationdate input formats straight through
- 06:08 PM Revision 638: test/input/SALVIAS_db.sh: Updated DB name
- 05:57 PM Revision 637: Regenerated vegbien.ERD exports
- 05:55 PM Revision 636: input.Makefile: Added documentation for why import errors for one input do not abort the import process for all inputs
- 05:53 PM Revision 635: input.Makefile: Determine DB name from input directory name, rather than DB file name
- 05:20 PM Revision 634: input.Makefile: Added documentation for accepting a test output
- 05:07 PM Revision 633: mappings/Makefile: Don't delete DwC-VegBIEN.specimens.csv in clean
- 05:05 PM Revision 632: VegBIEN: Renamed taxondetermination.*determination to is*
- 04:16 PM Revision 631: inputs/SALVIAS/test: Ignore test outputs
- 04:12 PM Revision 630: input.Makefile: Added test that generates VegBIEN.2-step.xml by mapping via a VegX.xml
- 03:39 PM Revision 629: input.Makefile: Added test that generates VegX.xml
- 03:29 PM Revision 628: input.Makefile: Added test that generates VegBIEN.xml
- 02:50 PM Revision 627: input.Makefile: Factored test/import.out out of DB section
- 02:43 PM Revision 626: input.Makefile: Renamed test/import.ref to import.out.ref. Changed syntax for accepting a test output to work with all types of test outputs.
- 02:32 PM Revision 625: Makefiles: Recurse into outermost subdir rather than bypassing it and going directly to innermost subdir
- 02:08 PM Revision 624: input.Makefile: Deal with inputs without a DB file, tests, verifications, etc.
- 02:07 PM Revision 623: inputs/Makefile: Don't use subdir makefiles because they are no longer needed
- 01:28 PM Revision 622: input.Makefile: Detect DB engine automatically from SQL file available in src subdir
- 01:00 PM Revision 621: input.Makefile: Factored as much as possible out of section for each DB engine
- 12:58 PM Revision 620: input.Makefile: Moved tests into test subdir
- 12:51 PM Revision 619: input.Makefile: Moved tests into test subdir
- 12:46 PM Revision 618: Added initial DwC-VegBIEN mappings spreadsheet with DwC terms
- 12:36 PM Revision 617: inputs/SALVIAS/verify: Updated to use new names for renamed tables
- 12:34 PM Revision 616: inputs/SALVIAS/verify: Updated to use new names for renamed tables
- 12:18 PM Revision 615: vegbien.sql: Removed no longer needed specimen.collector_id
- 12:16 PM Revision 614: VegX-VegBIEN mapping: Map collector name to new verbatimcollectorname field
- 12:11 PM Revision 613: vegbien.sql: Removed specimen.collectornumber_dwc and replaced it with verbatimcollectorname to reflect that the collectornumber_dwc is actually an ID of the specimen, and the collector's name is what we want to store
01/23/2012
- 05:33 PM Revision 612: mappings/Makefile: Run simplify_xpath on VegX-VegBIEN.organisms.csv
- 05:33 PM Revision 611: simplify_xpath: Be case sensitive to handle VegX correctly
- 05:18 PM Revision 610: VegX-VegBIEN mapping: Avoid using a dummy taxondetermination with role=collector
- 04:50 PM Revision 609: VegX-VegBIEN mapping: Map stem count to new stemcount field
- 04:48 PM Revision 608: VegX-VegBIEN mapping: Take advantage of aggregateoccurrence.count being optional
- 04:46 PM Revision 607: vegbien.sql: Made aggregateoccurrence.count to handle individuals data (for which count should be dynamically determined from # individual plants inside the aggregateoccurrence)
- 04:43 PM Revision 606: NYBG-VegBIEN mapping: Don't map dummy values to locationcode, etc. (e.g. in specimens data) because these tables are no longer required
- 04:42 PM Revision 605: vegbien.sql: Made several pointers to parent elements optional to deal with specimens data that might not have a location, etc.
- 04:29 PM Revision 604: vegbien.sql: Added taxondetermination UNIQUE constraint
- 04:18 PM Revision 603: VegX-VegBIEN mapping: Took advantage of location.confidentialitystatus being optional
- 04:16 PM Revision 602: VegX-VegBIEN mapping: Took advantage of userdefined.userdefinedtype being optional
- 04:13 PM Revision 601: vegbien.sql: Gave userdefined.userdefinedtype a default value
- 04:10 PM Revision 600: VegX-VegBIEN mappings: Took advantage of plantconcept.reference_id becoming optional
- 04:07 PM Revision 599: vegbien.sql: Made plantconcept.reference_id optional. Merge plantconcepts with no reference_id when eliminating duplicates.
- 04:05 PM Revision 598: PostgreSQL-MySQL.csv: Deal with all non-NOT NULL timestamp fields
- 04:00 PM Revision 597: vegbien.sql: Removed confusing plantconcept.plantname field since we are using plantname.plantname instead
- 03:54 PM Revision 596: VegBIEN: Renamed aux_role to role
- 03:47 PM Revision 595: VegX-VegBIEN mappings: Took advantage of several fields becoming optional
- 03:43 PM Revision 594: VegX-VegBIEN mappings: Took advantage of several fields becoming optional
- 03:36 PM Revision 593: Regenerated vegbien.ERD exports
- 03:36 PM Revision 592: vegbien.sql: taxonbinmethod points to stratumtype instead of stratum because stratumtype is a method table, but stratum is a measurements table. stratum does not point directly to stratummethod because it points to it via stratumtype.
- 03:04 PM Revision 591: vegbien.sql: Made taxondetermination.determinationdate optional because some determinations might not have a date
- 02:47 PM Revision 590: Regenerated vegbien.ERD exports
- 02:46 PM Revision 589: vegbien.sql: Added specimen.authorspecimencode
- 02:36 PM Revision 588: Adjusted vegbien.ERD.mwb
- 02:34 PM Revision 587: Regenerated vegbien.ERD exports
- 02:33 PM Revision 586: VegBIEN: Renamed sourceaccessionnumber to sourceaccessioncode to show that they are the data source's analog of accessioncode. Added sourceaccessioncode to all applicable tables because this is the database pkey, which is distinct from any author*code applied by the collector.
- 01:56 PM Revision 585: vegbien.sql: Changed taxonbinmethod_keys to UNIQUE INDEX to take advantage of COALESCE() for dealing with NULL values
- 01:43 PM Revision 584: vegbien.sql: Renamed taxonbin to taxonbinmethod to reflect that it does not contain actual organisms (those go in aggregateoccurrence), but rather defined a *method* of aggregating organisms
- 01:39 PM Revision 583: vegbien.sql: Removed taxonbin.count because that belongs in aggregateoccurrence and taxonbin is more similar to a sampling method. Added taxonbin UNIQUE constraint.
- 01:24 PM Revision 582: vegbien.sql: Do location duplicate elimination independently on code or lat/long, allowing duplicate entries with NULLs to exist when a location is incompletely specified
- 01:15 PM Revision 581: vegbien.sql: Require location to have either an authorlocationcode or a lat/long. Distinguish between regular and subplots in UNIQUE constraint.
- 01:10 PM Revision 580: vegbien.sql: Renamed location.latitude and longitude to publiclatitude, publiclongitude to reflect that they are not the actual lat/long. Switched to requiring reallatitude/reallongitude.
- 01:02 PM Revision 579: Added inputs/TurboVeg
- 12:58 PM Revision 578: vegbien.ERD.mwb: Deal with MySQL assuming that a timestamp field is NOT NULL
- 12:57 PM Revision 577: PostgreSQL-MySQL.csv: Deal with MySQL assuming that a timestamp field is NOT NULL
- 12:53 PM Revision 576: vegbien.sql: Made specimen.taxonoccurrence_id required
- 12:47 PM Revision 575: vegbien.sql: Made several fields optional, adding defaults where needed
- 12:47 PM Revision 574: PostgreSQL-MySQL.csv: Deal with PostgreSQL-style :: casts
- 12:14 PM Revision 573: NYBG mappings: Add mapping for CollectorNumber to specimen.collectornumber_dwc
- 12:08 PM Revision 572: vegbien.sql: Added specimen.collectornumber_dwc
- 12:03 PM Revision 571: VegBIEN: Renamed sourceid to author*code
- 12:01 PM Revision 570: VegBIEN: Renamed sourceid to author*code
- 11:44 AM Revision 569: mappings: Map ScientificNameAuthor to plantconcept with rank author
- 10:53 AM Revision 568: vegbien.sql: Removed sizeclass.*precision fields
- 10:50 AM Revision 567: vegbien.sql: Added count field to taxonBin
- 10:47 AM Revision 566: VegBIEN: Renamed place to locationplace
- 10:40 AM Revision 565: vegbien.sql.make: Exclude 'CREATE OR REPLACE PROCEDURAL LANGUAGE plpgsql' to avoid restore errors
- 10:20 AM Revision 564: repl: When matching words, treat _ as word boundary and ignore case.
01/20/2012
- 05:09 PM Revision 563: VegBIEN: Renamed individualplant.height to overallheight
- 03:02 PM Task #337 (Closed): VegBIEN table renames
- E-mail from Brad on 2012-1-20:
Hi Mike,
You and Aaron have convinced me. Bob & I were advocating a return to th... - 02:34 PM Revision 562: Regenerated vegbien.ERD exports
- 02:32 PM Revision 561: PostgreSQL-MySQL.csv: Convert double quotes to backticks (`)
- 02:10 PM Revision 560: Regenerated vegbien.ERD exports
- 02:09 PM Revision 559: vegbien.for_ERD.sql: Fixed syntax error in stratum table
- 02:00 PM Revision 558: Regenerated vegbien.ERD exports
- 01:39 PM Revision 557: vegbien.sql: Renamed sizeclass.*accuracy to *precision to reflect intended purpose
- 01:33 PM Revision 556: vegbien.sql: Added stem.diameteraccuracy
- 01:22 PM Revision 555: vegbien.sql: Removed taxonoccurrence.currentdetermination_id and originaldetermination_id because it's not possible to create a two-way pointer when the child-to-parent pointer is required (chicken-and-egg problem)
- 01:05 PM Revision 554: README.TXT: Added make empty_db
- 12:52 PM Revision 553: vegbien.sql: Added taxonoccurrence.currentdetermination_id and originaldetermination_id
- 12:41 PM Revision 552: vegbien.ERD.mwb: Added stratum and place tables
- 12:29 PM Task #338 (Resolved): put together samples of SALVIAS data that aggregates individuals and stems in different ways
- * individual count
* stem count
* individuals
* stems
01/19/2012
- 06:06 PM Task #337: VegBIEN table renames
- "observation" can be confusing, because it's not obvious what it's an observation of (a plot, an organism, a stem?). ...
- 06:02 PM Task #337 (Closed): VegBIEN table renames
- E-mail from Brad:
Conceptual definitions of tables:
observation: permanent attributes of a unit or area of obse... - 05:32 PM Task #336 (Resolved): create VegBIEN ERD
- Now live on [[VegBIEN schema]]!
- 04:56 PM Task #336: create VegBIEN ERD
- I discovered that MySQL Workbench can synchronize its model with an existing CREATE script. So, it looks like we can ...
- 04:54 PM Task #336 (Resolved): create VegBIEN ERD
- 05:28 PM Revision 551: Added vegbien.ERD.png
- 05:24 PM Revision 550: Added vegbien.ERD.svg
- 05:21 PM Revision 549: Added vegbien.ERD.pdf
- 05:19 PM Revision 548: Added vegbien ERD
- 05:19 PM Revision 547: Ignore *.bak in schemas
- 05:07 PM Revision 546: VegBIEN: specimen links directly to taxonoccurrence rather than indirectly via individualplant
- 04:43 PM Revision 545: vegbien.sql: Made voucher fields NOT NULL
- 04:41 PM Revision 544: vegbien.sql: Added voucher table
- 04:26 PM Revision 543: VegBIEN: specimen links directly to taxonoccurrence rather than indirectly via individualplant
- 04:22 PM Revision 542: VegBIEN: specimen links directly to taxonoccurrence rather than indirectly via individualplant
- 04:12 PM Revision 541: Create schemas/vegbien.for_wiki.sql from schemas/vegbien.for_ERD.sql
- 04:11 PM Revision 540: Added lib/ERD-wiki.csv to convert ERD SQL to human-readable wiki format
- 04:10 PM Revision 539: schemas/vegbien.for_ERD.sql: Removed VegBank-internal fields from taxonoccurrence
- 03:51 PM Revision 538: Removed no longer needed schemas/vegbien.my.sql
- 03:51 PM Revision 537: PostgreSQL-MySQL.csv: Use \g<0> instead of \0 in replacements because \0 not supported. Replace fulltext with fulltext__ (note fulltext_ doesn't work).
- 01:34 PM Revision 536: Added schemas/vegbien.for_ERD.sql and generated MySQL
- 01:34 PM Revision 535: Regenerated schemas/vegbien.for_ERD.sql
- 01:33 PM Revision 534: PostgreSQL-MySQL.csv: Deal with serial NOT NULL fields
- 01:22 PM Revision 533: Generate MySQL DDL schemas/vegbien.my.sql from vegbien.sql
- 01:22 PM Revision 532: Added repl spreadsheet to map PostgreSQL DDL to MySQL
- 12:56 PM Revision 531: Ignore src subdirectories in inputs
- 12:46 PM Revision 530: VegBIEN: Use stem instead of individualplant to store diameter/dbh
- 11:35 AM Task #302 (Resolved): Make changes to VegBIEN schema
- Extensive changes have been made
- 11:33 AM Task #331 (Resolved): use VegX's individualOrganismObservation/relatedItem to model stems
- Stems are now modeled in VegX. (Note that _VegBIEN_ stems import not complete yet)
- 11:32 AM Task #319: Update statistics/lists of user-defined fields in use in VegX and VegBIEN
- I've done this for NYBG to VegX
- 11:31 AM Task #322 (Resolved): get access to raw CTFS data
- I have the data from Shash, including a SQL dump
- 11:30 AM Task #334 (Resolved): which DarwinCore attributes do we want to include in the specimen table?
- Desired fields have been added or moved from other tables
- 11:30 AM Task #314: Import CTFS data
- I have data from Shash and am working with her to make sure we can parse it
- 11:29 AM Task #318 (Resolved): decide whether to switch to VegX 1.5.2
- In fact, we will switch to 1.5.3, which is now available
01/18/2012
- 05:57 PM Revision 529: vegbien.sql: Removed no longer needed taxondetermination.plantname_id
- 05:53 PM Revision 528: VegBIEN: Made taxondetermination.plantconcept_id required and plantname_id optional
- 04:28 PM Revision 527: vegbien.sql: Added specimen.taxonoccurrence_id fkey
- 03:47 PM Revision 526: vegbien.sql: Removed no longer needed taxondetermination.individualplant_id, specimen_id fields
- 01:57 PM Revision 525: VegBIEN: Renamed plot to location to reflect that a location can also be used for specimens
- 01:29 PM Revision 524: vegbien.sql: Added individualplant.stemcount
- 12:38 PM Revision 523: vegbien.sql: Added accessioncode fields to existing, modified tables
- 12:21 PM Revision 522: vegbien.sql: Added accessioncode fields to new tables
01/17/2012
- 04:56 PM Revision 521: vegbien.sql: Removed no longer needed individualplant.taxonarea
- 04:40 PM Revision 520: VegBIEN: Moved taxondetermination.sourceaccessionnumber and museum_id to specimen
- 03:32 PM Revision 519: vegbien.sql: Split taxondetermination UNIQUE CONSTRAINT into one constraint for each main key (taxonoccurrence_id, specimen_id, individualplant_id) to deal with Postgres treating a row with NULL in any of those columns as distinct. Added CHECK constraint to require one of taxonoccurrence_id, specimen_id, individualplant_id to be NOT NULL.
- 02:47 PM Revision 518: VegBIEN: Renamed museumaccessionnumber to sourceaccessionnumber
- 02:42 PM Revision 517: VegBIEN: Renamed authorcode to sourceid
- 02:33 PM Revision 516: vegbien.sql: Updated sizeclass to use ranges for parameters and remove taxonarea
01/13/2012
- 05:25 PM Revision 515: input.Makefile: Fixed verify by making %.out: % rule use %.make instead
- 05:24 PM Revision 514: input.Makefile: Moved verifications into one set of verify.* files
- 05:18 PM Revision 513: input.Makefile: All tests are run in the same transaction, with output to test.out
- 05:04 PM Revision 512: bin/map: Added support for processing multiple map_paths at once in the same transaction
- 04:50 PM Revision 511: SALVIAS organisms mapping: Removed redundant PlotCode mapping because the association to plotevent is done with PlotID
- 04:49 PM Revision 510: bin/map: Started adding support for multiple map_paths, all run in the same db transaction
- 04:35 PM Revision 509: Filled in SALVIAS stems mappings. make test2's stems test fails because we don't wrap the import transaction around all the tests so that organisms are already in the database when stems are imported.
- 04:28 PM Revision 508: mappings/Makefile: Corrected stemobservation to stem
- 03:46 PM Revision 507: Use baseDistance/complexUserDefined/method to distinguish between different dbhs in VegX
- 03:41 PM Revision 506: VegX mappings: Use baseDistance/value instead of baseDistance so we can later use complexUserDefined to distinguish between different types of dbh
- 03:28 PM Task #335 (Rejected): Look into using Sybase Powerbuilder or IBM Enterprise Vision to map data
- The TACC (Texas Advanced Computing Center) people might have individual licenses they could let us use
- 03:07 PM Revision 505: VegX-VegBIEN.organisms.csv: Added initial stems mappings
- 03:06 PM Revision 504: mappings/Makefile: Updated VegX-VegBIEN.stems.csv input/output roots
- 03:05 PM Revision 503: ch_root: Deal with roots followed by something other than \b
- 02:41 PM Revision 502: input.Makefile: Remove *.out in new verify and test subdirs
- 02:38 PM Revision 501: Moved inputs files into subdirs for maps, verify, and test operations
- 02:00 PM Revision 500: vegbien.sql: Removed taxondetermination.collector_id because that's now in specimen
- 01:58 PM Revision 499: VegBIEN: Map to specimen.collector_id instead of taxondetermination
- 01:27 PM Revision 498: vegbien.sql: Added specimen.collector_id column
- 01:20 PM Revision 497: vegbien.sql: Changed taxondetermination.stem_id to individualplant_id
- 01:16 PM Revision 496: vegbien.sql: Removed taxondetermination columns that have been moved to specimen
- 01:14 PM Revision 495: VegBIEN: Merged specimen.collectionnumber and catalognumber_dwc fields. Removed NOT NULL constraints on several specimen columns because not all specimens have all identifiers. Mapped to new specimen fields.
- 12:59 PM Revision 494: VegX-VegBIEN mapping: Map collectiondate to specimen table
- 12:55 PM Revision 493: vegbien.sql: Added specimen.collectionnumber and collectiondate fields
- 12:50 PM Revision 492: Updated README.TXT to reflect new make targets
01/12/2012
- 04:40 PM Task #334 (Resolved): which DarwinCore attributes do we want to include in the specimen table?
- 04:37 PM Task #314: Import CTFS data
- e-mails on 2012-1-12:
Steve Dolins:
We used the identical aggregate data file used for BIEN I... I think the da... - 04:30 PM Task #333 (Resolved): add traits table to VegBIEN
- 04:25 PM Task #318: decide whether to switch to VegX 1.5.2
- Brad: Yes! Latest version please, 1.5.2. It's about to get a lot less stable in any case, as we make more changes.
- 04:10 PM Task #318 (Resolved): decide whether to switch to VegX 1.5.2
- We have been mapping to VegX 1.5.1 (the latest stable version) but a release candidate VegX 1.5.2 is also available. ...
- 04:24 PM Task #332 (Rejected): switch to VegX 1.5.3
- 04:22 PM Task #331 (Resolved): use VegX's individualOrganismObservation/relatedItem to model stems
- 04:22 PM Task #330 (Rejected): DwC extension to VegX
- 04:22 PM Task #329 (New): make website user-protected
- 04:21 PM Task #328 (New): get raw data rather than flat file exports from RAINFOR
- 04:21 PM Task #327 (Resolved): look into Clio
- 04:21 PM Task #326 (Rejected): generic MOU template to request data
- 04:21 PM Task #325 (Resolved): map CTFS data
- 04:21 PM Task #324 (Resolved): NYBG validations
- 04:20 PM Task #323 (Resolved): map NYBG data
- 04:20 PM Task #322 (Resolved): get access to raw CTFS data
- 04:20 PM Task #321 (Resolved): Convert user-defined VegBIEN fields to first-class fields
- 04:20 PM Task #320 (Rejected): Convert user-defined VegX fields to first-class fields
- 04:20 PM Task #319 (Resolved): Update statistics/lists of user-defined fields in use in VegX and VegBIEN
- 01:25 PM Task #302: Make changes to VegBIEN schema
- I have applied most of Bob's changes to the VegBIEN schema, as well as many of Mark's suggestions. I posted a summary...
- 01:18 PM Revision 491: vegbien.sql: Added sizeclass columns
- 01:01 PM Revision 490: VegBIEN: Removed stem* prefix from stem measurements because it's redundant and because these same field names were reused in non-stem tables
- 12:51 PM Revision 489: VegBIEN: Renamed stemobservation to stem
- 12:49 PM Revision 488: vegbien.sql: Added comments to renamed tables specifying VegBank equivalent
- 12:45 PM Revision 487: vegbien.sql: Added comments to new tables
- 12:25 PM Revision 486: vegbien.sql: Removed r# prefix from foreign key constraints
- 12:21 PM Revision 485: vegbien.sql: Added plot.namedplace_id for theplots which are identified by a namedplace rather than lat/long or name
- 12:18 PM Revision 484: vegbien.sql: Made plot.authorcode optional as not all plots have names
- 12:16 PM Task #317 (Rejected): Direct mapping from VegX to VegBIEN
- When complete, this will be a complete, generic mapping of all elements in VegX to all corresponding elements in VegB...
- 12:15 PM Revision 483: vegbien.sql: Added _dwc to specimen columns that are from DwC
- 12:12 PM Revision 482: vegbien.sql: Added UNIQUE constraint for specimen table. Made catalognumber required so that all specimens can be uniquely identified.
- 12:09 PM Revision 481: vegbien.sql: Made specimen.individualplant_id optional because specimens are not required to have a corresponding plot event (which all individualplants must have)
- 12:05 PM Revision 480: VegBIEN-VegBank.csv: Updated to reflect renames
- 12:02 PM Revision 479: vegbien.sql: Renamed specimen.code to collectioncode to match DwC term
- 11:58 AM Revision 478: vegbien.sql: Added sizeclass table
- 11:51 AM Revision 477: vegbien.sql: Renamed collection to specimen to reflect that it's for a single collection specimen
- 11:45 AM Revision 476: VegBIEN: Renamed collectiveobservation to aggregateoccurrence to reflect VegX usage of aggregate and the new taxonoccurrence table
01/11/2012
- 02:47 PM Revision 475: VegBIEN: Renamed stemobservation.stemcode to authorcode
- 02:43 PM Revision 474: vegbien.sql: Renamed individualplant.code to authorcode
- 02:41 PM Revision 473: sql.py: Reordered get() args to match put()
- 02:40 PM Revision 472: db_xml.py: Use new sql.put() which only eliminates duplicates based on database UNIQUE constraints
- 02:38 PM Revision 471: sql.py: Added new function put() and changed get() to use it
- 02:32 PM Revision 470: sql.py: Fixed raising of NullValueException to pass array of columns rather than single column
- 02:25 PM Revision 469: vegbien.sql: Added UNIQUE constraints to tables used in the mapping so that duplicates are eliminated by the database
- 02:23 PM Revision 468: sql.py: ExceptionWithColumn now stores multiple columns so that they can all be used in DuplicateKeyExceptions
- 02:22 PM Revision 467: util.py: Fixed dict_subset() to work correctly when a key does not exist
- 02:13 PM Revision 466: util.py: Added dict_subset()
- 02:07 PM Revision 465: sql.py: Use constraint_cols() to determine DuplicateKeyException column in try_insert()
- 01:50 PM Revision 464: sql.py: Added constraint_cols() to get columns of a constraint for use in determining the columns for a DuplicateKeyException
- 11:59 AM Revision 463: db_xml.py: Map empty fields to None instead of ignoring them, to ensure that an existing row with a value in that field is not used in the duplicate elimination
- 11:56 AM Revision 462: bin/map: Use db_xml.put()'s new on_error handler
- 11:53 AM Revision 461: db_xml.py: put(): Added on_error handler. Reordered arguments according to how often they are likely to be used.
- 11:31 AM Revision 460: bin/map: Print # input rows processed
- 11:08 AM Task #314: Import CTFS data
- e-mail from Brad Boyle on 2012-1-10:
Interesting observation. I must admit, I haven't inspected Shash's VegX yet, ...
Also available in: Atom