Project

General

Profile

Activity

From 01/16/2012 to 02/14/2012

02/14/2012

06:57 PM Revision 1013: xpath.py: get(): Fixed bug where it would try to create a node named . or .. if . or .. didn't have matching attributes. Now it will just reuse the current or parent node, but create any needed attrs if create is True.
Aaron Marcuse-Kubitza
06:52 PM Revision 1012: util.py: Added list_eq_is() to compare two lists using is
Aaron Marcuse-Kubitza
06:23 PM Revision 1011: xpath.py: Don't allow rooted attributes (doesn't make sense), in case someone tries to do elem[/rooted_attr]
Aaron Marcuse-Kubitza
05:52 PM Revision 1010: bin/map: Moved root.clear() into separate function prep_root() that can be called whenever needed
Aaron Marcuse-Kubitza
05:43 PM Revision 1009: xpath.py: Added get() support for references (different from pointers) to dynamically set the value of an attribute
Aaron Marcuse-Kubitza
05:21 PM Revision 1008: util.py: Added list_get()
Aaron Marcuse-Kubitza
05:10 PM Revision 1007: util.py: Added is_list()
Aaron Marcuse-Kubitza
05:00 PM Revision 1006: bin/map: Use var doc0_root for quick reference to doc0's root
Aaron Marcuse-Kubitza
04:57 PM Revision 1005: xpath.py: get(): Go to root when empty element is encountered at the beginning of an XPath. Added allow_rooted parameter to turn off this functionality when XPaths with a leading slash should not be considered rooted.
Aaron Marcuse-Kubitza
04:32 PM Revision 1004: xpath.py: Don't consider a path starting with "." to be rooted. Do this by not automatically translating an empty path name to ".".
Aaron Marcuse-Kubitza
04:18 PM Revision 1003: xpath.py: Added is_rooted()
Aaron Marcuse-Kubitza
04:16 PM Revision 1002: xpath.py: Added elem_is_empty()
Aaron Marcuse-Kubitza
04:10 PM Revision 1001: xpath.py: Added documentation labels to each section
Aaron Marcuse-Kubitza
04:04 PM Revision 1000: xpath.py: Added support for getting the parent node when encountering ".."
Aaron Marcuse-Kubitza
04:02 PM Revision 999: xml_dom.py: Added parent() to get parent node without recursing past the root node to the document object. Documented that NodeParentIter incorporates this sanity check.
Aaron Marcuse-Kubitza
03:44 PM Revision 998: xpath.py: get(): Renamed parent to root to better reflect that it's the starting point for the search. Calling it parent will later be confusing when we want to get the parent node using "..".
Aaron Marcuse-Kubitza
03:31 PM Revision 997: xpath.py: Added parser support for attribute values that are references to another part of the XML tree
Aaron Marcuse-Kubitza
03:21 PM Revision 996: xml_func.py: Fixed module description comment to reflect that not all XML funcs generate text
Aaron Marcuse-Kubitza
03:20 PM Revision 995: xml_func.py: Refactored to add funcs to the module funcs variable as they are defined. Renamed defined functions to the name of the corresponding XML function.
Aaron Marcuse-Kubitza
03:01 PM Revision 994: xml_func.py: Added _ignore func to "comment out" an XML subtree
Aaron Marcuse-Kubitza
02:55 PM Revision 993: input.Makefile: Fixed error message when no DB file found so that it doesn't incorrectly imply that PostgreSQL inputs are supported
Aaron Marcuse-Kubitza
02:54 PM Revision 992: input.Makefile: Fixed error message when no DB file found so that it doesn't incorrectly imply that PostgreSQL inputs are supported
Aaron Marcuse-Kubitza
02:42 PM Revision 991: input.Makefile: Don't run tests in verbose mode because the run time stats, etc. are not relevant
Aaron Marcuse-Kubitza
02:41 PM Revision 990: bin/map: Only print error/run time stats in verbose mode. input.Makefile: Run import in verbose mode so that error/run time stats are still printed.
Aaron Marcuse-Kubitza
02:36 PM Revision 989: Moved value to string conversion functions infrom util.py to new module format.py
Aaron Marcuse-Kubitza
02:32 PM Revision 988: exc.py, profiling.py: Use util.int2str() to print # iters with thousands separators
Aaron Marcuse-Kubitza
02:32 PM Revision 987: util.py: Added int2str()
Aaron Marcuse-Kubitza
02:26 PM Revision 986: bin/map: Document that the exit status is the # of errors in the import, up to the maximum exit status
Aaron Marcuse-Kubitza
02:24 PM Revision 985: exc.py: Generalize ExTracker to not just print the # of errors at exit. Instead, provide an exit() method that the ExTracker creator can call at exit to set the exit status to the # of errors. This fixes the Python bug where a benign error message was printed if SystemExit was raised in an atexit function.
Aaron Marcuse-Kubitza
02:15 PM Revision 984: bin/map: Set ExPercentTracker's iter_text. Start ExPercentTracker after input processing, because errors in command line options should just end the program and don't need to be tracked.
Aaron Marcuse-Kubitza
02:13 PM Revision 983: exc.py: ExPercentTracker: Added ability to set custom iter_text, similar to ItersProfiler
Aaron Marcuse-Kubitza
02:07 PM Revision 982: bin/map: Use profiling.ItersProfiler. Refactored input row count calculation to have each function aggregate and return the row count, and then display the row count and statistics that depend on it at the end of the program.
Aaron Marcuse-Kubitza
02:06 PM Revision 981: Added profiling.py to time operations and provide the user with statistical information
Aaron Marcuse-Kubitza
02:03 PM Revision 980: util.py: Added basic to_si() to add SI prefix to value
Aaron Marcuse-Kubitza
01:02 PM Revision 979: util.py: Added format_str() to use locale-specific formatting settings, including thousands separator. Use it in to_percent().
Aaron Marcuse-Kubitza
12:48 PM Revision 978: bin/map: Use new ExPercentTracker to print error rate (% of # rows) when program exits
Aaron Marcuse-Kubitza
12:47 PM Revision 977: exc.py: Added ExPercentTracker to track errors as % of iterations
Aaron Marcuse-Kubitza
12:47 PM Revision 976: util.py: Added to_percent()
Aaron Marcuse-Kubitza
12:07 PM Revision 975
Aaron Marcuse-Kubitza
12:02 PM Revision 974: exc.py: print_ex(): Declare emph param as a keywork param instead of popping it from **format
Aaron Marcuse-Kubitza
11:54 AM Revision 973: exc.py: print_ex(): Declare emph param as a keywork param instead of popping it from **format
Aaron Marcuse-Kubitza

02/13/2012

04:38 PM Revision 972: inputs/SALVIAS/maps/VegX.organisms.csv: Mapped OrigSpecies and OrigGenus combined to new plantlevel Binomial
Aaron Marcuse-Kubitza
04:35 PM Revision 971: xpath.py: Fixed bug where value of XPath (used for copying to other branches) is retrieved after first XPath element is popped rather than before, which can sometimes leave an empty XPath for value() to run on
Aaron Marcuse-Kubitza
03:33 PM Revision 970: mappings/DwC-VegBIEN.specimens.csv: Fixed bien.vegetation mapping to point to commconcept->commname. Fixed bien.substrate mapping to point to locationevent.landscapenarrative.
Aaron Marcuse-Kubitza
03:25 PM Revision 969: inputs/NYBG/maps/DwC.specimens.csv: Mapped CoordinatePrecision using _noCV
Aaron Marcuse-Kubitza
03:24 PM Revision 968: xml_func.py: Added _noCV func to check that non-ratio-scale data does not contain CV values
Aaron Marcuse-Kubitza
02:34 PM Revision 967: mappings/DwC-VegBIEN.specimens.csv: Fixed locality fields mapping to go to location.locationnarrative
Aaron Marcuse-Kubitza
02:05 PM Revision 966: input.Makefile: For all input types, including DB, import each table in a separate map invocation
Aaron Marcuse-Kubitza
01:58 PM Revision 965: xml_func.py: _range: Treat a None from or to value as an unknown (a la SQL NULL) and return None instead of raising a SyntaxException
Aaron Marcuse-Kubitza
01:57 PM Revision 964: xml_dom.py: NodeTextEntryIter: Convert empty entries (including entries containing error comments) to None
Aaron Marcuse-Kubitza
01:48 PM Revision 963: xml_dom.py: replace(): Added support for new node that's None (deletes existing node)
Aaron Marcuse-Kubitza
12:59 PM Revision 962: xml_func.py: Put SyntaxException's cause on same line as error message so that the whole error is treated as distinct by error_stats
Aaron Marcuse-Kubitza
12:47 PM Revision 961: Added errors_filter_before and errors_filter_after to prepare `map` error messages for easy filtering and then restore line breaks
Aaron Marcuse-Kubitza
12:46 PM Revision 960: error_stats: Fixed to work on Mac
Aaron Marcuse-Kubitza
11:26 AM Revision 959: error_stats: Simplified to use uniq --count option
Aaron Marcuse-Kubitza
11:21 AM Revision 958: input.Makefile: Print error message if no input file found (for file input type). This fixes a bug where map would just take input from stdin when no input file redirect or input DB env vars were specified.
Aaron Marcuse-Kubitza
11:10 AM Revision 957: map: Map standard DB names to original DB names on nimoy
Aaron Marcuse-Kubitza
10:40 AM Task #358: make shortlist of 1st-level fields a method should have
e-mail from Brad on 2012-2-1:
* plot shape: rectangle, circle, line
* area sampled: e.g. 1.0 ha
* plot x: e.g. 100 m...
Aaron Marcuse-Kubitza
10:29 AM Revision 956: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
10:26 AM Revision 955: vegbien.sql: Added methodtrait and methodtraitname tables
Aaron Marcuse-Kubitza
10:24 AM Revision 954: PostgreSQL-MySQL.csv: Handle array types
Aaron Marcuse-Kubitza
09:54 AM Revision 953: vegbien.ERD.mwb: Recolored plant tables to all have the same color, distinct from the occurrence color
Aaron Marcuse-Kubitza
09:32 AM Revision 952: mappings/VegX-VegBIEN.organisms.csv: Added mappings for SALVIAS fields with no join mapping. This fixes the last of the "no join mapping" errors.
Aaron Marcuse-Kubitza
09:27 AM Revision 951: input.Makefile: svn_props: Set svn:ignore on maps subdirs
Aaron Marcuse-Kubitza
09:26 AM Revision 950: inputs/SALVIAS-CSV/maps/VegX.plots.csv: Fixed mappings without a join mapping in VegX-VegBIEN.*.csv
Aaron Marcuse-Kubitza
09:20 AM Revision 949: VegX mappings: Gentry DBH mapping: Use VegX's attribute and method tables
Aaron Marcuse-Kubitza
09:05 AM Revision 948: mappings/VegX-VegBIEN.organisms.csv: Removed no longer used mapping to taxondetermination.determinationdate. This also prevents ever creating a taxondetermination without a plantconcept.
Aaron Marcuse-Kubitza

02/11/2012

07:31 PM Revision 947: bin/map: Added redo option to control whether the database is emptied before inserting new data. Can be used to turn off emptying the DB in test mode, because this is often slow and is not needed if you are running tests on an empty testing database.
Aaron Marcuse-Kubitza
07:25 PM Revision 946: opts.py: env_flag(): Added support for default value if unset
Aaron Marcuse-Kubitza
07:25 PM Revision 945: opts.py: env_flag(): Added support for default value if unset
Aaron Marcuse-Kubitza
07:16 PM Revision 944: bin/map: Use env_flag()'s new env_names usage support to print flags usage
Aaron Marcuse-Kubitza
07:16 PM Revision 943: opts.py: Added env_names usage support to env_flag()
Aaron Marcuse-Kubitza
07:05 PM Revision 942: mappings/VegX-VegBIEN.organisms.csv: Removed no longer needed mapping for taxonDetermination/note
Aaron Marcuse-Kubitza
07:03 PM Revision 941: inputs/SALVIAS-CSV/maps/VegX.organisms.csv: Map cfaff to taxonConcept/fit, which maps to taxondetermination.taxonFit
Aaron Marcuse-Kubitza
06:59 PM Revision 940: inputs/SALVIAS/maps/VegX.organisms.csv: Map cfaff to taxonConcept/fit, which maps to taxondetermination.taxonFit
Aaron Marcuse-Kubitza
06:39 PM Revision 939: join: Print a warning if no join mapping found (in addition to adding this warning to the comments column)
Aaron Marcuse-Kubitza
06:27 PM Revision 938: Removed no longer needed inputs/NYBG/maps/VegX.organisms.csv because NYBG is now mapped via DwC
Aaron Marcuse-Kubitza
06:26 PM Revision 937: mappings/VegX-VegBIEN.organisms.csv: Removed mappings used only by NYBG, because NYBG now maps via DwC
Aaron Marcuse-Kubitza
06:11 PM Revision 936: Added ch_root_via to transform a map spreadsheet to use a different root, using a connecting root that links the input and output roots together
Aaron Marcuse-Kubitza
06:02 PM Revision 935: Added cols to select columns from a spreadsheet
Aaron Marcuse-Kubitza
06:02 PM Revision 934: util.py: Added list_subset()
Aaron Marcuse-Kubitza
05:42 PM Revision 933: ch_root: Fixed detection of unset env vars so that usage message is printed when any option is missing
Aaron Marcuse-Kubitza
05:41 PM Revision 932: opts.py: Call an error handler if an env var isn't set
Aaron Marcuse-Kubitza
05:40 PM Revision 931: util.py: Added function wrappers for statements noop() and and_()
Aaron Marcuse-Kubitza
05:07 PM Revision 930: inputs/NYBG-CSV: Map via DwC
Aaron Marcuse-Kubitza
04:50 PM Revision 929: Added subtract to subtract map spreadsheets
Aaron Marcuse-Kubitza
04:49 PM Revision 928: ch_root: Ignore empty lines
Aaron Marcuse-Kubitza
04:06 PM Revision 927: Added intersect to intersect two map spreadsheets
Aaron Marcuse-Kubitza
03:48 PM Revision 926: union: Clarified overwrite order of inputs in description
Aaron Marcuse-Kubitza
03:31 PM Revision 925: Removed no longer needed mappings/review
Aaron Marcuse-Kubitza
03:31 PM Revision 924: mappings/Makefile: Regenerate for_review maps automatically when a map changes
Aaron Marcuse-Kubitza
03:17 PM Revision 923: mappings/review: Generalized to convert all mappings to VegBIEN, not just a specific listed set (which was out of date)
Aaron Marcuse-Kubitza
03:16 PM Revision 922: mappings/for_review/DwC-VegBIEN.specimens.csv: Regenerated
Aaron Marcuse-Kubitza
03:12 PM Revision 921: inputs/NYBG/maps/DwC.specimens.csv: Fixed CollectedDate mapping to use the _date XML func
Aaron Marcuse-Kubitza
03:07 PM Revision 920: DwC mappings: Mapped Substrate and Vegetation
Aaron Marcuse-Kubitza
02:54 PM Revision 919: DwC mappings: Mapped BoundingBox, footprintWKT to location.dsgpoly
Aaron Marcuse-Kubitza
02:44 PM Revision 918: DwC mappings: Mapped Notes and PlantFungusDescription to bien.specimenDescription, merged together
Aaron Marcuse-Kubitza
02:41 PM Revision 917: xml_func.py: Added _merge and _label XML funcs
Aaron Marcuse-Kubitza
02:12 PM Revision 916: vegbien.sql: Added specimenreplicate.description
Aaron Marcuse-Kubitza
02:01 PM Revision 915: DwC mappings: Corrected mappings for CollectorNumber, individualID, recordNumber, and fieldNumber
Aaron Marcuse-Kubitza
01:31 PM Revision 914: DwC mappings: Mapped infraspecificEpithet to/from Subspecies
Aaron Marcuse-Kubitza
01:22 PM Revision 913: DwC-VegBIEN mapping: merge decimalLatitude/Longitude with verbatimLatitude/Longitude
Aaron Marcuse-Kubitza

02/10/2012

06:03 PM Task #346: add georeferencing support to schema
We will add a separate locationdetermination for the GNRS-validated place, with calculated set to true. Aaron Marcuse-Kubitza
06:01 PM Task #355 (Resolved): implement morphospecies using a scoped plantconcept
Aaron Marcuse-Kubitza
02:40 PM Task #355 (Resolved): implement morphospecies using a scoped plantconcept
Aaron Marcuse-Kubitza
06:01 PM Task #356 (Resolved): support subplot events that link to parent plot events
Aaron Marcuse-Kubitza
02:41 PM Task #356 (Resolved): support subplot events that link to parent plot events
Aaron Marcuse-Kubitza
05:55 PM Revision 912: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
05:53 PM Revision 911: vegbien.ERD.mwb: Fixed lines. Repositioned elements.
Aaron Marcuse-Kubitza
05:41 PM Revision 910: vegbien.sql: Added locationevent.parent_id for subplot events to point to their parent plot event
Aaron Marcuse-Kubitza
05:21 PM Revision 909: vegbien.sql: Added plantconceptscope.namedplace_id
Aaron Marcuse-Kubitza
05:14 PM Revision 908: vegbien.sql: Added taxonoccurrence.method_id
Aaron Marcuse-Kubitza
05:00 PM Revision 907: vegbien.sql: Renamed plotmethod to method
Aaron Marcuse-Kubitza
04:47 PM Revision 906: vegbien.sql: Added locationdetermination.identifier_id
Aaron Marcuse-Kubitza
04:39 PM Revision 905: vegbien.ERD.mwb: Fixed lines
Aaron Marcuse-Kubitza
04:35 PM Revision 904: VegBIEN: Renamed locationplace to locationdetermination
Aaron Marcuse-Kubitza
04:18 PM Revision 903: vegbien.sql: Removed morphospecies, instead using plantconcept and new table plantconceptscope to represent scoped morphospecies. Reorganized plant* tables in ERD.
Aaron Marcuse-Kubitza
03:41 PM Revision 902: vegbien.ERD.mwb: Added plantcorrelation table to ERD
Aaron Marcuse-Kubitza
02:43 PM Task #357 (Rejected): make shortlist of 1st-level fields a method should have
duplicate of #358 Aaron Marcuse-Kubitza
02:42 PM Task #357 (Rejected): make shortlist of 1st-level fields a method should have
(Brad and Mike)
* area, size, etc.
* see example plot methods on nimoy in @/home/bien_shared/methodology/plotMeth...
Aaron Marcuse-Kubitza
02:42 PM Task #358 (Resolved): make shortlist of 1st-level fields a method should have
(Brad and Mike)
* area, size, etc.
* see example plot methods on nimoy in @/home/bien_shared/methodology/plotMeth...
Aaron Marcuse-Kubitza

02/09/2012

07:08 PM Revision 901: mappings/VegX-VegBIEN.organisms.csv: Fixed TaxonomicRankBelowSubspeciesEnum mapping to use plantname instead of placing the name directly inside plantconcept. This bug was probably left over from the switch from linking directly to plantname rather than to a plantconcept which has a plantname.
Aaron Marcuse-Kubitza
06:54 PM Revision 900: Renamed get_errors to error_stats and added counts of each error message
Aaron Marcuse-Kubitza
06:27 PM Task #352: create way to represent methods hierarchically in schema
I added a basic plotmethod table. What hierarchical information do we want to add? Aaron Marcuse-Kubitza
06:26 PM Task #350: implement methods in VegBIEN
I added a basic plotmethod table Aaron Marcuse-Kubitza
06:24 PM Revision 899: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
06:22 PM Revision 898: vegbien.sql: Added plotmethod table
Aaron Marcuse-Kubitza
05:42 PM Task #348: 1st draft of schema
e-mail from Martha Narro on 2012-2-9:
* 1:1 constraint on taxonOccurrence has been dropped, so that topic is final...
Aaron Marcuse-Kubitza
05:40 PM Task #345: integrate GNRS into VegBIEN
See "iPlant Geospatial Roadmap":https://pods.iplantcollaborative.org/wiki/display/SOT/Geospatial+Roadmap+%28revised%2... Aaron Marcuse-Kubitza
05:36 PM Task #346: add georeferencing support to schema
"DwC georeferencing fields":http://code.google.com/p/darwincore/wiki/Location#Georeference_Terms:
* decimalLatitude
...
Aaron Marcuse-Kubitza
05:34 PM Task #296 (Resolved): Direct mapping from native salvias_plots MySQL database to VegBIEN
SALVIAS stems have been mapped. Mappings are in SVN under "@inputs/SALVIAS/maps@":https://projects.nceas.ucsb.edu/nce... Aaron Marcuse-Kubitza
05:31 PM Task #304: Complete full dataset imports to VegBIEN via VegX of NYBG and SALVIAS
First full NYBG load is complete, with 0.12% error rate. Full stats under [[Benchmarking]]. Aaron Marcuse-Kubitza
05:30 PM Task #313 (Resolved): Import full NYBG data
First full NYBG load is complete, with 0.12% error rate. Full stats under [[Benchmarking]]. Aaron Marcuse-Kubitza
05:29 PM Task #323 (Resolved): map NYBG data
First full NYBG load is complete, with 0.12% error rate. Full stats under [[Benchmarking]]. Aaron Marcuse-Kubitza
05:28 PM Task #342 (Resolved): make DwC_BIEN schema
[[DwC schema]] has a list of fields in DwC-BIEN, mappings from different versions of DwC, and mappings to VegBIEN Aaron Marcuse-Kubitza

02/07/2012

05:22 PM Revision 897: input.Makefile: Renamed $(test) variable (containing the command to run a test) so it wouldn't conflict with test environment variable
Aaron Marcuse-Kubitza
05:10 PM Revision 896: mappings to VegBIEN: Don't map notes to taxondetermination because some organisms have notes but no species name
Aaron Marcuse-Kubitza
05:08 PM Revision 895: mappings/Makefile: Fixed bug in rules for cleaning up core maps whenever they change, which had the target and prerequisite reversed
Aaron Marcuse-Kubitza
04:39 PM Revision 894: input.Makefile: nolog option defaults to on when test is on
Aaron Marcuse-Kubitza
04:36 PM Revision 893: input.Makefile: Fixed bug where no log file was being created, even when nolog was off
Aaron Marcuse-Kubitza
04:11 PM Revision 892: Replaced all type(...) == str with util.is_str(...) to properly treat Unicode objects as strings
Aaron Marcuse-Kubitza
04:08 PM Revision 891: xml_dom.py: minidom.Element.write_opening(): Use new Attr.__str__() method for printing attributes
Aaron Marcuse-Kubitza
04:06 PM Revision 890: bin/map: For XML inputs, use xpath.get()'s new multiple result support to iterate over elements matching the root, rather than just iterating over the first matching element's siblings. This fixes the broken 2-step tests for stems, which didn't filter by the root's attributes.
Aaron Marcuse-Kubitza
04:04 PM Revision 889: xpath.py: get(): Fixed bug where limit was not passed to recursive get() calls
Aaron Marcuse-Kubitza
04:02 PM Revision 888: xml_dom.py: by_tag_name(): Iterate forwards over children unless last_only optimization turned on. Added Attr.__str__() and __repr__() for debug-printing Attrs.
Aaron Marcuse-Kubitza
03:10 PM Revision 887: xpath.py: get(): Turn off last_only optimization when limit > 1
Aaron Marcuse-Kubitza
03:07 PM Revision 886: xpath.py: get(): Added full support for returning multiple matches
Aaron Marcuse-Kubitza
02:33 PM Revision 885: xpath.py: get(): Added basic structure for returning multiple matches. Added limit parameter to select one or many matches.
Aaron Marcuse-Kubitza
02:13 PM Revision 884: input.Makefile: test/VegBIEN.%.2-step.xml: Use the core map specific to the test's table instead of the main core map
Aaron Marcuse-Kubitza
02:10 PM Revision 883: bin/map: Print error if map root not found inXML input
Aaron Marcuse-Kubitza
01:52 PM Revision 882: mappings/: Removed mappings used by old tester
Aaron Marcuse-Kubitza
01:48 PM Revision 881: inputs/SALVIAS/maps/VegX.*.csv: Replaced symlinks with actual files
Aaron Marcuse-Kubitza
01:45 PM Revision 880: Removed old tester
Aaron Marcuse-Kubitza
01:44 PM Revision 879: Switched to using inputs/test as main test target
Aaron Marcuse-Kubitza
01:38 PM Revision 878: Added SALVIAS-CSV tests
Aaron Marcuse-Kubitza
01:31 PM Revision 877: Added NYBG-CSV tests
Aaron Marcuse-Kubitza
01:28 PM Revision 876: input.Makefile: Run separate tests for each map spreadsheet (input table) rather than all tables at once. This will make it possible to test CSV inputs, which have one CSV per table.
Aaron Marcuse-Kubitza
12:54 PM Revision 875: Added NYBG-CSV input
Aaron Marcuse-Kubitza
12:47 PM Revision 874: inputs/Makefile: Fixed forwarding of empty targets to subdirs
Aaron Marcuse-Kubitza
12:20 PM Revision 873: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
12:17 PM Revision 872: vegbien.sql: Added morphospecies table
Aaron Marcuse-Kubitza
11:53 AM Revision 871: vegbien.ERD.mwb: Fixed lines
Aaron Marcuse-Kubitza
11:45 AM Revision 870: vegbien.sql: Removed the taxonOccurrence:aggregateOccurrence 1:1 constraint
Aaron Marcuse-Kubitza

02/06/2012

05:22 PM Revision 869: sql.py: truncate(): Use run_raw_query() instead of run_query() because truncate() does not use the recover functionality of run_query(). Also, in the profiling output, this separates the "normal" SQL statements (which use run_query()) from the "core" SQL statements (which use run_raw_query()).
Aaron Marcuse-Kubitza
05:09 PM Revision 868: vegbien.sql: Added indexes for each field in party used in duplicate elimination (for use by sql.put()'s DuplicateKeyException handler)
Aaron Marcuse-Kubitza
04:58 PM Revision 867: sql.py: run_raw_query(): In debug mode, print query after params have been substituted in
Aaron Marcuse-Kubitza
04:55 PM Revision 866: sql.py: Fixed index_cols() to handle UNIQUE indexes with expressions, whose column names are stored in a different format
Aaron Marcuse-Kubitza
04:04 PM Revision 865: sql.py: Print warning if SELECT statement missing a WHERE, LIMIT, or OFFSET clause. Changed bin/map DB input get-all-rows statement to pass start=0 to suppress this warning for that statement.
Aaron Marcuse-Kubitza
03:51 PM Revision 864: db_xml.py: Added start option to get() that passes through to sql.select()
Aaron Marcuse-Kubitza
03:50 PM Revision 863: sql.py: Added start option to select() to set the OFFSET
Aaron Marcuse-Kubitza
03:27 PM Revision 862: sql.py: If run_raw_query.debug flag is set, print each query executed (on a single line)
Aaron Marcuse-Kubitza
03:25 PM Revision 861: strings.py: Added one_line() function to make a string all on one line
Aaron Marcuse-Kubitza
03:20 PM Revision 860: strings.py: Renamed one_line() to remove_extra_newl() to better reflect what it does
Aaron Marcuse-Kubitza
03:19 PM Revision 859: bin/map: Don't print Done after an action in debug logging mode because it messes up newlines when more debugging info is printed right after it
Aaron Marcuse-Kubitza
02:45 PM Revision 858: input.Makefile: Added nolog option to disable creating a log file, e.g. for debugging runs
Aaron Marcuse-Kubitza
01:57 PM Revision 857: xml_dom.py: Remove extra newlines from single-line strings (bin/map doesn't need to do this itself anymore)
Aaron Marcuse-Kubitza
01:56 PM Revision 856: strings.py: Added is_multiline() and one_line() for removing extra newlines from single-line strings
Aaron Marcuse-Kubitza
01:48 PM Revision 855: bin/map: In debug mode, print input XPath's XML tree all on one line
Aaron Marcuse-Kubitza
01:22 PM Revision 854: sql.py: Switched try_insert() to use index_cols() instead of constraint_cols() for "duplicate key value violates unique constraint" errors because they can also be generated by UNIQUE indexes (and there is a UNIQUE index for every UNIQUE constraint)
Aaron Marcuse-Kubitza
01:14 PM Revision 853: sql.py: Added index_cols() to get cols used by an index (similar to constraint_cols())
Aaron Marcuse-Kubitza
01:00 PM Revision 852: vegbien.sql: Fixed duplicate elimination for party to use a UNIQUE index with COALESCE for nullable fields
Aaron Marcuse-Kubitza
01:00 PM Revision 851: sql.py: Fixed bug in try_insert() where DuplicateKeyException was passed only cols[0] instead of cols array
Aaron Marcuse-Kubitza
12:49 PM Revision 850: vegbien.ERD.mwb: Fixed lines
Aaron Marcuse-Kubitza
12:30 PM Revision 849: Added get_errors to select just the error messages from `map` output
Aaron Marcuse-Kubitza
11:59 AM Revision 848: Added profile_stats to analyze a profiling statistics file
Aaron Marcuse-Kubitza
11:59 AM Revision 847: bin/map: Added profile_to option which turns on profiling to the specified file
Aaron Marcuse-Kubitza
11:13 AM Revision 846: bin/map: Added "if __name__ == '__main__': main()" idiom so file can be included as well as run. This will be useful for profiling.
Aaron Marcuse-Kubitza
11:12 AM Revision 845: dates.py: Fixed strftime() to pad years and days with leading zeros as datetime.strftime() does
Aaron Marcuse-Kubitza
11:01 AM Revision 844: dates.py: Work around strftime() bug that can't deal with 2/29 on a leap year
Aaron Marcuse-Kubitza
10:44 AM Revision 843: xml_func.py: Added FormatException for SyntaxExceptions generated by strftime() (which are often Python bugs)
Aaron Marcuse-Kubitza
10:25 AM Task #317: Direct mapping from VegX to VegBIEN
Mappings have been made for SALVIAS VegX fields Aaron Marcuse-Kubitza
10:23 AM Task #333: add traits table to VegBIEN
VegBank had an existing trait table, which was not included in the ERD. We can modify that table for our needs. Aaron Marcuse-Kubitza
10:23 AM Task #332: switch to VegX 1.5.3
VegX 1.5.3 is available, so we're switching to that instead Aaron Marcuse-Kubitza
10:22 AM Task #313: Import full NYBG data
Almost done with the first full load of the NYBG data. Currently takes 2 days. Aaron Marcuse-Kubitza
09:46 AM Revision 842: Added schemas/vegbank.revised.sql. Initial version has all "character varying" types replaced with text.
Aaron Marcuse-Kubitza
09:45 AM Revision 841: vegbien.sql: Replaced all "character varying" types with text, removing the length limits. Note that in PostgreSQL, text and "character varying" are stored the same way internally, so this does not affect performance or indexes.
Aaron Marcuse-Kubitza

02/04/2012

05:28 PM Revision 840: xml_dom.py: Added documentation labels to each section
Aaron Marcuse-Kubitza
05:21 PM Revision 839: xml_dom.py: Fixed bug in NodeTextEntryIter where an entry containing an element instead of a text node would be returned as the whole entry, instead of the value *of* the entry
Aaron Marcuse-Kubitza
04:56 PM Revision 838: bin/map: Added support for starting import at a specific row. Refactored row-processing code with and without a map to use a common process_rows() function (with the previous process_rows() being renamed to map_rows()).
Aaron Marcuse-Kubitza
04:14 PM Revision 837: bin/map: Use new util.cast()
Aaron Marcuse-Kubitza
04:14 PM Revision 836: util.py: Added cast() to cast a value while passing None through
Aaron Marcuse-Kubitza
04:09 PM Revision 835: bin/map: Print row # of rows with errors
Aaron Marcuse-Kubitza
03:27 PM Task #349: make ERD of full schema
Full ERD is available at [[VegBIEN schema]]. A few tables are not yet included, because their lines to existing table... Aaron Marcuse-Kubitza
03:25 PM Task #354 (Resolved): list of deliverables for next week (2/6-2/10)
See [[2012-02-03 conference call]] Aaron Marcuse-Kubitza
03:22 PM Revision 834: sql.py: Fixed error in pkey() where recover was not passed as a named parameter to run_query()
Aaron Marcuse-Kubitza
03:12 PM Revision 833: sql.py: Added documentation labels to each section
Aaron Marcuse-Kubitza
03:10 PM Revision 832: sql.py: Added documentation labels to each section
Aaron Marcuse-Kubitza
03:03 PM Revision 831: db_xml.py: Used new sql.py recover functionality
Aaron Marcuse-Kubitza
03:03 PM Revision 830: sql.py: Added ability to recover from database errors so you don't get the error "InternalError: current transaction is aborted, commands ignored until end of transaction block"
Aaron Marcuse-Kubitza
02:25 PM Revision 829: vegbien.sql: Removed taxonoccurrence.taxoninferencearea because it's duplicated in aggregateoccurrence.inferencearea
Aaron Marcuse-Kubitza
02:18 PM Revision 828: bin/map: Highlight the "input row" and "output row:" labels in error messages
Aaron Marcuse-Kubitza
02:17 PM Revision 827: xml_func.py: Highlight nodes that were commented out because of errors
Aaron Marcuse-Kubitza
02:17 PM Revision 826: exc.py: Print exceptions with the first line highlighted in red
Aaron Marcuse-Kubitza
02:16 PM Revision 825: term.py: Added emph() and error()
Aaron Marcuse-Kubitza
01:51 PM Revision 824: vegbien.ERD.mwb: Fixed lines
Aaron Marcuse-Kubitza
01:47 PM Revision 823: vegbien.ERD.mwb: Fixed lines
Aaron Marcuse-Kubitza
01:46 PM Revision 822: vegbien.ERD.mwb: Added note, notelink, and revision
Aaron Marcuse-Kubitza
01:43 PM Revision 821: vegbien.ERD.mwb: Added embargo to diagram
Aaron Marcuse-Kubitza
01:27 PM Revision 820: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
01:24 PM Revision 819: vegbien.ERD.mwb: Fixed lines. Added "Core subset" and "Other tables" labels.
Aaron Marcuse-Kubitza
01:03 PM Revision 818: xml_func.py: Changed _date func to use new dates.strftime(), which can handle years before 1900
Aaron Marcuse-Kubitza
01:03 PM Revision 817: Added dates.py to handle date/time manipulation, such as fixing Python's broken strftime() that can't handle years before 1900
Aaron Marcuse-Kubitza
01:02 PM Revision 816: Regenerated mappings/for_review/DwC-VegBIEN.specimens.csv
Aaron Marcuse-Kubitza

02/03/2012

06:15 PM Revision 815: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
06:12 PM Revision 814: vegbien.ERD.mwb: Added reference and party tables
Aaron Marcuse-Kubitza
05:49 PM Revision 813: vegbien.ERD.mwb: Fixed lines
Aaron Marcuse-Kubitza
05:48 PM Revision 812: filter_ERD.csv: Remove fkeys to heavily-linked tables (reference, party)
Aaron Marcuse-Kubitza
05:33 PM Task #347 (Resolved): revise milestones
Brad and Martha have approved it Aaron Marcuse-Kubitza
03:15 PM Task #347: revise milestones
My revisions are on the wiki under *[[Milestones]]* Aaron Marcuse-Kubitza
01:57 PM Task #347 (Resolved): revise milestones
Aaron Marcuse-Kubitza
05:30 PM Revision 811: Added to_do/milestones.doc
Aaron Marcuse-Kubitza
05:27 PM Revision 810: Renamed milestones.doc to timeline.doc
Aaron Marcuse-Kubitza
05:24 PM Revision 809: Added schemas/filter_ERD.csv and use it when generating vegbien.my.sql
Aaron Marcuse-Kubitza
05:16 PM Revision 808: vegbien.ERD.mwb: Added cover* to main diagram
Aaron Marcuse-Kubitza
04:53 PM Revision 807: vegbien.ERD.mwb: Started adding additional tables "below the fold" on the 2nd page
Aaron Marcuse-Kubitza
04:34 PM Revision 806: vegbien.ERD.mwb: Fixed lines
Aaron Marcuse-Kubitza
04:26 PM Revision 805: vegbien.ERD.mwb: Moved legend to top left to make room for more misc tables. Organized legend by location on diagram.
Aaron Marcuse-Kubitza
04:14 PM Revision 804: vegbien.ERD.mwb: Added soilobs table
Aaron Marcuse-Kubitza
04:07 PM Revision 803: vegbien.ERD.mwb: Added userdefined tables. Fixed lines.
Aaron Marcuse-Kubitza
03:57 PM Revision 802: vegbien.ERD.mwb: Changed location color to match VegBank ERD
Aaron Marcuse-Kubitza
03:53 PM Revision 801: vegbien.ERD.mwb: Added trait to diagram
Aaron Marcuse-Kubitza
03:38 PM Revision 800: vegbien.ERD.mwb: Added plantstatus to diagram. Added margins around diagram.
Aaron Marcuse-Kubitza
03:18 PM Task #320: Convert user-defined VegX fields to first-class fields
Brad's meeting notes on 2012-2-3:
* Aaron to make whatever modifications are needed to match VegBien schema
* Sub...
Aaron Marcuse-Kubitza
02:49 PM Revision 799: Added milestones.doc
Aaron Marcuse-Kubitza
02:04 PM Task #346: add georeferencing support to schema
* support DwC georeferencing fields Aaron Marcuse-Kubitza
01:57 PM Task #346 (New): add georeferencing support to schema
Aaron Marcuse-Kubitza
02:03 PM Task #354 (Resolved): list of deliverables for next week (2/6-2/10)
Aaron Marcuse-Kubitza
02:02 PM Task #353 (Resolved): add terms from previous versions of DwC to DwC-BIEN
Aaron Marcuse-Kubitza
02:02 PM Task #352 (Resolved): create way to represent methods hierarchically in schema
Aaron Marcuse-Kubitza
02:01 PM Task #351 (Resolved): list of fields and method attributes needed to know whether can combine data from different plots
* critical methodological elements Aaron Marcuse-Kubitza
02:00 PM Task #350 (Resolved): implement methods in VegBIEN
* middle ground between descriptions and ontological metadata Aaron Marcuse-Kubitza
01:59 PM Task #349 (New): make ERD of full schema
Aaron Marcuse-Kubitza
01:58 PM Task #348 (Resolved): 1st draft of schema
* plots and specimens data Aaron Marcuse-Kubitza
01:56 PM Task #345 (Resolved): integrate GNRS into VegBIEN
* geovalidation in scope
* georeferencing out of scope unless existing implementation
Aaron Marcuse-Kubitza
01:56 PM Task #344 (Resolved): contact John about GNRS/georeferencing
* Using PostGIS/R? Aaron Marcuse-Kubitza
01:55 PM Task #343 (Resolved): integrate TNRS into VegBIEN
Aaron Marcuse-Kubitza
12:39 PM Revision 798: DwC mappings: Fixed syntax of _date XML funcs to not wrap dates twice in a _date func
Aaron Marcuse-Kubitza
12:38 PM Revision 797: xml_func.py: Fixed bug in SyntaxException constructor where the cause was not passed to ExceptionWithCause
Aaron Marcuse-Kubitza
12:24 PM Revision 796: xml_dom.py: Override Node.__repr__ and Element.__repr__ to make sure self.toprettyxml() is used in all cases where a Node is converted to a string
Aaron Marcuse-Kubitza
11:57 AM Revision 795: util.py: Don't print type name in ConstraintError message because sometimes an entry tuple (e.g. from a dict) is checked, and then the type isn't useful
Aaron Marcuse-Kubitza
11:53 AM Revision 794: xml_func.py: Use conv_items(str, ...) in XML funcs that require strings, but don't otherwise use conv_items()
Aaron Marcuse-Kubitza
11:50 AM Revision 793: xml_func.py: Use xml_dom.TextEntryOnlyIter to check inputs for XML funcs that only handle strings, not whole elements
Aaron Marcuse-Kubitza
11:49 AM Revision 792: xml_dom.py: Added TextEntryOnlyIter to check that values returned by NodeTextEntryIter are strings from text nodes
Aaron Marcuse-Kubitza
11:48 AM Revision 791: util.py: Added is_str() and CheckedIter
Aaron Marcuse-Kubitza

01/31/2012

07:52 PM Revision 790: input.Makefile: Log each import to a new log file named according to the current time
Aaron Marcuse-Kubitza
07:38 PM Revision 789: xml_func.py: SyntaxException: Use exc.ExceptionWithCause
Aaron Marcuse-Kubitza
07:37 PM Revision 788: exc.py: Added str_() to convert an Exception to a string. Use str_() in print_ex().
Aaron Marcuse-Kubitza
07:31 PM Revision 787: xml_func.py: conv_items(): Only trap type conversion errors, not errors in map_items(). A previous bug incorrectly interpreted an error caught by map_items() as a SyntaxException.
Aaron Marcuse-Kubitza
07:23 PM Revision 786: xml_func.py: _date func: Fixed bug where setting defaults of a date part and removing zeros needed to happen after elements were converted to int. Refactored to use conv_items().
Aaron Marcuse-Kubitza
06:58 PM Revision 785: Makefiles: Added documentation labels to each section
Aaron Marcuse-Kubitza
06:55 PM Revision 784: Makefile: Added documentation labels to each section
Aaron Marcuse-Kubitza
06:49 PM Revision 783: input.Makefile: Added documentation labels to each section
Aaron Marcuse-Kubitza
06:44 PM Revision 782: xml_dom.py: Fixed conversion of minidom.Node to a string to use toprettyxml() again
Aaron Marcuse-Kubitza
06:40 PM Revision 781: Added inputs/SALVIAS-CSV/maps/VegBIEN.* maps
Aaron Marcuse-Kubitza
06:39 PM Revision 780: input.Makefile: If a test doesn't yet have accepted output, print its current output
Aaron Marcuse-Kubitza
06:24 PM Revision 779: mappings/Makefile: Only clean up a core map (intermediate format->VegBIEN) when it's modified. Note that the .last_cleanup files need to be under version control to prevent cleanups upon running any map command in a new checkout.
Aaron Marcuse-Kubitza
06:07 PM Revision 778: xml_func.py: Support dates with year/month/day == 0
Aaron Marcuse-Kubitza
06:06 PM Revision 777: input.Makefile: Print informative message if test failure is ignored
Aaron Marcuse-Kubitza
05:47 PM Revision 776: input.Makefile: Print path to test output so you can tell which directory a test is running in
Aaron Marcuse-Kubitza
05:45 PM Revision 775: input.Makefile: Fixed 2-step tests, which were still using $(root)/map when all bin/map calls were supposed to be routed through $(map)
Aaron Marcuse-Kubitza
05:05 PM Revision 774: xml_dom.py: Print simple XML elements on one line
Aaron Marcuse-Kubitza
04:33 PM Revision 773: Added inputs/SALVIAS-CSV/maps
Aaron Marcuse-Kubitza
04:33 PM Revision 772: input.Makefile: Added import support for CSV and XML inputs (no test support yet)
Aaron Marcuse-Kubitza
03:48 PM Revision 771: inputs/SALVIAS-CSV: Added src files
Aaron Marcuse-Kubitza
03:47 PM Revision 770: inputs/SALVIAS-CSV: Added src files
Aaron Marcuse-Kubitza
03:42 PM Revision 769: inputs: Removed no longer needed svn:ignore
Aaron Marcuse-Kubitza
03:41 PM Revision 768: input.Makefile: Coded svn:ignore directly in Makefile
Aaron Marcuse-Kubitza
03:40 PM Revision 767: inputs/Makefile: Added %-add target to add new inputs
Aaron Marcuse-Kubitza
03:35 PM Revision 766: input.Makefile: Added svn_props rule for updating svn:ignore of dir and subdirs
Aaron Marcuse-Kubitza
02:42 PM Revision 765: vegbien.sql: Made aggregateoccurrence:taxonoccurrence relationship 1:1
Aaron Marcuse-Kubitza
02:03 PM Revision 764: Added inputs/CTFS/src, with subset of files that will be in version control
Aaron Marcuse-Kubitza
02:01 PM Revision 763: mappings/for_review/DwC-VegBIEN.specimens.csv: Updated
Aaron Marcuse-Kubitza

01/30/2012

07:14 PM Revision 762: inputs/NYBG: Map via DwC instead of VegX
Aaron Marcuse-Kubitza
07:12 PM Revision 761: mappings/DwC-VegBIEN.specimens.csv: Provide schema URLs for namespaces (in the column header)
Aaron Marcuse-Kubitza
07:11 PM Revision 760: input.Makefile: Deal with core maps (the main *-VegBIEN map) whose roots are other than organisms
Aaron Marcuse-Kubitza
07:09 PM Revision 759: bin/map: In the xml_func.process() for XML outputs, pass an on_error that outputs to ex_tracker.track()
Aaron Marcuse-Kubitza
07:08 PM Revision 758: xpath.py: set_id(): Deal with XPaths with an
Aaron Marcuse-Kubitza
07:07 PM Revision 757: xml_dom.py: replace_with_text(): Accept Node inputs for cases where e.g. an XML func with an error has been replaced with a comment
Aaron Marcuse-Kubitza
07:06 PM Revision 756: xml_func.py: _date: Year defaults to 1900
Aaron Marcuse-Kubitza
05:18 PM Revision 755: DwC-VegBIEN.specimens.csv: Output DwC as XML instead of CSV
Aaron Marcuse-Kubitza
05:05 PM Revision 754: xpath.py: Allow XPath names to be quoted literals, just like values. Handle : embedded in a name (even without quotes), often indicating a namespace.
Aaron Marcuse-Kubitza
04:55 PM Revision 753: Removed no longer needed lib/ERD-wiki.csv
Aaron Marcuse-Kubitza
04:54 PM Revision 752: Parser.py: In syntax errors, only print <END> at end of string if error was *at* end of string
Aaron Marcuse-Kubitza
04:50 PM Revision 751: bin/map: Print error message rather than assertion failure for CSV output, which isn't supported yet
Aaron Marcuse-Kubitza
04:47 PM Revision 750: input.Makefile: Don't abort tester if via test fails, because CSV output is not supported yet
Aaron Marcuse-Kubitza
04:15 PM Task #323: map NYBG data
Infrastructure exists to do full NYBG loads, but there are many errors in the import Aaron Marcuse-Kubitza
04:12 PM Task #338 (Resolved): put together samples of SALVIAS data that aggregates individuals and stems in different ways
See [[SALVIAS issues]] Aaron Marcuse-Kubitza
04:02 PM Revision 749: input.Makefile: Deal properly with more than one format that's being mapped via (e.g. DwC and VegX)
Aaron Marcuse-Kubitza
03:49 PM Revision 748: Renamed mappings/DwC-VegBIEN.organisms.csv back to mappings/DwC-VegBIEN.specimens.csv because it now uses /specimenreplicate as the output root
Aaron Marcuse-Kubitza
03:43 PM Revision 747: Renamed mappings/DwC-VegBIEN.organisms.csv back to mappings/DwC-VegBIEN.specimens.csv because it now uses /specimenreplicate as the output root
Aaron Marcuse-Kubitza
03:41 PM Revision 746: mappings/DwC-VegBIEN.organisms.csv: Fixed syntax error in a mapping
Aaron Marcuse-Kubitza
03:35 PM Revision 745: mappings/DwC-VegBIEN.organisms.csv: Sorted
Aaron Marcuse-Kubitza
03:34 PM Revision 744: mappings/Makefile: Sort DwC-VegBIEN.organisms.csv
Aaron Marcuse-Kubitza
03:33 PM Revision 743: inputs/NYBG/maps/VegBIEN.organisms.csv: Regenerated
Aaron Marcuse-Kubitza
03:12 PM Revision 742: mappings/DwC-VegBIEN.organisms.csv: Sort on output column
Aaron Marcuse-Kubitza
03:04 PM Revision 741: Renamed mappings/DwC-VegBIEN.specimens.csv to DwC-VegBIEN.organisms.csv because it (currently) uses /taxonoccurrence as the output root
Aaron Marcuse-Kubitza
03:03 PM Revision 740: Renamed mappings/DwC-VegBIEN.specimens.csv to DwC-VegBIEN.organisms.csv because it (currently) uses /taxonoccurrence as the output root
Aaron Marcuse-Kubitza
03:01 PM Revision 739: Makefile: Support recursing into subdir when no target is specified (e.g. make mappings/)
Aaron Marcuse-Kubitza
02:55 PM Revision 738: union: Merge rows (and column labels) using maps.merge_mappings()
Aaron Marcuse-Kubitza
02:51 PM Revision 737: join: Merge the column labels as well
Aaron Marcuse-Kubitza
02:51 PM Revision 736: maps.py: Eliminate duplicates when merging values in the same column
Aaron Marcuse-Kubitza
02:06 PM Revision 735: join: Moved mappings-specific merge functionality into maps.merge_mappings()
Aaron Marcuse-Kubitza
01:55 PM Revision 734: join: Use merge_rows() from new maps.py
Aaron Marcuse-Kubitza
01:54 PM Revision 733: Added new library maps.py for map spreadsheet manipulation
Aaron Marcuse-Kubitza
01:38 PM Revision 732: join: Merge comments of input map and join map
Aaron Marcuse-Kubitza
01:37 PM Revision 731: join: Report which input mappings are missing a mapping in the join map
Aaron Marcuse-Kubitza
01:03 PM Revision 730: join: Report which input mappings are missing a mapping in the join map
Aaron Marcuse-Kubitza
12:28 PM Revision 729: inputs/NYBG/maps/VegX.organisms.csv: Added note that the primary key has NULL values in some rows
Aaron Marcuse-Kubitza
12:24 PM Revision 728: input.Makefile: Preserve as many intermediate files as possible (make likes to delete intermediates if they aren't marked as .PRECIOUS)
Aaron Marcuse-Kubitza
11:46 AM Revision 727: sort: Sort empty strings last so that inputs with no mapping go at the end of the map spreadsheet
Aaron Marcuse-Kubitza
11:24 AM Revision 726: VegBIEN-VegBank.csv: Updated for recent table renames
Aaron Marcuse-Kubitza
11:21 AM Revision 725: join: For input mappings with no match in the join map, include them in the output map with an empty mapping
Aaron Marcuse-Kubitza
11:05 AM Revision 724: input.Makefile: Generalized to handle mapping via any format, not just VegX
Aaron Marcuse-Kubitza
10:44 AM Revision 723: input.Makefile: Don't print message to accept output for failing 2-step tests, because they use another test's accepted output
Aaron Marcuse-Kubitza

01/28/2012

06:02 PM Revision 722: input.Makefile: Don't abort tester if only 2-step test fails, as it's often finicky
Aaron Marcuse-Kubitza
05:55 PM Revision 721: xml_func.py: Raise xml_func.SyntaxException for ValueErrors generated by date.strftime() (e.g. year out of range due to poor Y2K support in some OS implementations of strftime)
Aaron Marcuse-Kubitza
05:51 PM Revision 720: xml_func.py: Raise xml_func.SyntaxException for ValueErrors generated by datetime.date() (e.g. month out of range)
Aaron Marcuse-Kubitza
05:46 PM Revision 719: vegbien.sql: Added project.reference_id to namespace project names by datasource
Aaron Marcuse-Kubitza
05:39 PM Revision 718: input.Makefile: Import all tables at once by default
Aaron Marcuse-Kubitza
05:11 PM Revision 717: bin/map: Print "Inserted ... new rows into database" message to stdout rather than stderr so it can be stored in the test case output as a validation check
Aaron Marcuse-Kubitza
05:08 PM Revision 716: Accepted initial test output for NYBG/test/import.out.ref
Aaron Marcuse-Kubitza
05:06 PM Revision 715: bin/map: Clean up datasource input values
Aaron Marcuse-Kubitza
05:06 PM Revision 714: strings.py: Added std_newl() to convert line endings and cleanup() to process strings with extra or nonstandard whitespace
Aaron Marcuse-Kubitza
04:46 PM Revision 713: PostgreSQL-MySQL.csv: Deal with custom types
Aaron Marcuse-Kubitza
04:36 PM Revision 712: vegbien.sql: Added aggregateoccurrence.occurrencestatus_dwc field
Aaron Marcuse-Kubitza
04:22 PM Revision 711: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
04:21 PM Revision 710: vegbien.ERD.mwb: Added commclass table to ERD
Aaron Marcuse-Kubitza
04:11 PM Revision 709: vegbien.sql: Removed direct pointer from location to namedplace because locationplace already has this relationship and we don't want to have an extra pointer just for duplicate elimination
Aaron Marcuse-Kubitza
03:56 PM Revision 708: vegbien.ERD.mwb: Added stratummethod to ERD
Aaron Marcuse-Kubitza
03:53 PM Revision 707: vegbien.sql: Removed locationevent.stratummethod_id because the stratummethod is a per-stratum (or technically, per-stratumtype) field
Aaron Marcuse-Kubitza
03:28 PM Revision 706: PostgreSQL-MySQL.csv: Remove CHECK constraints
Aaron Marcuse-Kubitza
03:20 PM Revision 705: PostgreSQL-MySQL.csv: Remove functions and triggers
Aaron Marcuse-Kubitza
03:13 PM Revision 704: vegbien.sql: Ensure that aggregateoccurrence.count == 1 when the aggregateoccurrence has a plantobservation. Use a trigger to do this automatically.
Aaron Marcuse-Kubitza
02:51 PM Revision 703: README.TXT: Added command for reimporting data
Aaron Marcuse-Kubitza
02:48 PM Revision 702: README.TXT: Added instructions to sync ERD with vegbien.sql schema. Organized commands into categories.
Aaron Marcuse-Kubitza
02:23 PM Revision 701: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
02:22 PM Revision 700: Added BIEN_logo.png
Aaron Marcuse-Kubitza
02:19 PM Revision 699: vegbien.ERD.mwb: Added color group legend
Aaron Marcuse-Kubitza
01:59 PM Revision 698: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
01:58 PM Revision 697: vegbien.ERD.mwb: Fixed lines
Aaron Marcuse-Kubitza
01:57 PM Revision 696: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
01:57 PM Revision 695: vegbien.ERD.mwb: Fixed lines and moved plant to its own color category
Aaron Marcuse-Kubitza
01:52 PM Revision 694: vegbien.ERD.mwb: Added colors to ERD
Aaron Marcuse-Kubitza
01:34 PM Revision 693: vegbien.ERD.mwb: Simplified diagram by removing column types
Aaron Marcuse-Kubitza
12:56 PM Revision 692: schemas/Makefile: Don't generate for_ERD DDLs because the ERD is now synced with the full schema
Aaron Marcuse-Kubitza
12:54 PM Revision 691: vegbien.ERD.mwb: Synced with whole schema
Aaron Marcuse-Kubitza
12:54 PM Revision 690: vegbien.sql: Reordered fields in tables truncated in the ERD so that all removed fields are at the end of the table
Aaron Marcuse-Kubitza
12:44 PM Revision 689: schemas/Makefile: Generate MySQL version of vegbien.sql as well as vegbien.for_ERD.sql for eventual use in syncing the ERD with the whole schema
Aaron Marcuse-Kubitza
12:43 PM Revision 688: PostgreSQL-MySQL.csv: Added translations for syntaxes used by pg_dump
Aaron Marcuse-Kubitza
12:41 PM Revision 687: PostgreSQL-MySQL.csv: Added translations for syntaxes used by pg_dump
Aaron Marcuse-Kubitza
12:40 PM Revision 686: repl: All regexps are by default in multiline and ignore case mode
Aaron Marcuse-Kubitza

01/27/2012

04:59 PM Revision 685: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
04:59 PM Revision 684: vegbien.sql: Made planttag a child of plantobservation instead of plant, since tags change over time
Aaron Marcuse-Kubitza
04:39 PM Revision 683: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
04:37 PM Revision 682: vegbien.sql: Removed no longer used plantobservation.aggregateoccurrence_id
Aaron Marcuse-Kubitza
04:35 PM Revision 681: VegX-VegBIEN mapping: Link aggregateoccurrence to plantobservation via forward pointer rather than backward child-to-parent pointer
Aaron Marcuse-Kubitza
04:32 PM Revision 680: vegbien.sql: Made plantobservation.aggregateoccurrence_id optional because link will soon be going in the other direction
Aaron Marcuse-Kubitza
04:25 PM Revision 679: vegbien.sql: Removed taxonbinmethod table since its fields are now in aggregateoccurrence
Aaron Marcuse-Kubitza
04:14 PM Revision 678: vegbien.sql: Added taxonbinmethod fields to aggregateoccurrence
Aaron Marcuse-Kubitza
04:10 PM Revision 677: vegbien.sql: Added back aggregateoccurrence.stratum_id
Aaron Marcuse-Kubitza
04:05 PM Revision 676: vegbien.sql: Added stratum.area
Aaron Marcuse-Kubitza
04:03 PM Revision 675: vegbien.sql: Removed denormalized duplicate fields from stratum
Aaron Marcuse-Kubitza
03:58 PM Revision 674: vegbien.sql: Added plant and planttag tables
Aaron Marcuse-Kubitza
03:43 PM Revision 673: VegBIEN: Renamed stem to stemobservation
Aaron Marcuse-Kubitza
03:35 PM Revision 672: vegbien.sql: Removed specimenreplicate:taxonoccurrence 1:1 requirement
Aaron Marcuse-Kubitza
03:32 PM Revision 671: VegBIEN: Renamed individualplant to plantobservation
Aaron Marcuse-Kubitza
03:22 PM Revision 670: vegbien.sql: Updated table comments for specimenreplicate and specimen
Aaron Marcuse-Kubitza
03:19 PM Revision 669: vegbien.sql: Added specimen table to tie specimenreplicates together
Aaron Marcuse-Kubitza
03:12 PM Revision 668: VegBIEN: Renamed specimen to specimenreplicate
Aaron Marcuse-Kubitza
03:06 PM Revision 667: Remerged ERD DDL into ERD
Aaron Marcuse-Kubitza
03:04 PM Task #342 (Resolved): make DwC_BIEN schema
Aaron Marcuse-Kubitza
03:03 PM Task #341 (New): create verbal descriptions of edge cases
test cases for whether model can appropriately accommodate data Aaron Marcuse-Kubitza
02:55 PM Revision 666: Redoing commit that linked aggregateoccurrence forward to individualplant, allowing many taxonoccurrences (e.g. one for each specimen) to point to the same plant (e.g. that those specimens came from)
Aaron Marcuse-Kubitza
12:57 PM Revision 665: Added NYBG input
Aaron Marcuse-Kubitza
12:48 PM Revision 664: input.Makefile: Run tests with verbose output
Aaron Marcuse-Kubitza
12:48 PM Revision 663: bin/map: Fixed bug where verbose/debug flags were ignored and message were always printed.
Aaron Marcuse-Kubitza
12:39 PM Revision 662: bin/map: Added verbose and debug options. Added initial debug info.
Aaron Marcuse-Kubitza
12:39 PM Revision 661: xml_dom.py: Added is_simple() to determine whether every child recursively has no more than one child. Used is_simple() to print condensed XML when simple nodes are converted to a string.
Aaron Marcuse-Kubitza
11:58 AM Revision 660: vegbien.sql: Enforce 1:1 relationship between aggregateoccurrence<->individualplant and taxonoccurrence<->specimen
Aaron Marcuse-Kubitza
11:44 AM Revision 659: vegbien.sql: Changed individualplant UNIQUE constraint to enforce 1:1 relationship between aggregateoccurrence and individualplant
Aaron Marcuse-Kubitza
11:35 AM Revision 658: Undoing previous commit since it would prevent a plant from being tied to a data source, because the aggregateoccurrence pointer goes in the wrong direction
Aaron Marcuse-Kubitza
11:18 AM Revision 657: vegbien.sql: Added aggregateoccurrence.individualplant_id to make a 1:1 relationship between aggregateoccurrence and individualplant
Aaron Marcuse-Kubitza
01:55 AM Revision 656: input.Makefile: Generate VegBIEN.2-step.xml correctly from VegX.xml, by removing DB config env vars passed to map for that test case. Note that this causes the VegBIEN.2-step.xml test to fail, because the 2-step mapping does not yet match the 1-step mapping.
Aaron Marcuse-Kubitza
01:28 AM Revision 655: input.Makefile: Don't need to filter test output since stderr now goes to the screen
Aaron Marcuse-Kubitza
01:21 AM Revision 654: input.Makefile: Don't save *.err outputs for each test because this information is printed to the screen
Aaron Marcuse-Kubitza

01/26/2012

06:50 PM Revision 653: input.Makefile: Send echoed diff command to stdout of the make process (set -x echoes it to stderr)
Aaron Marcuse-Kubitza
06:48 PM Revision 652: input.Makefile: Write test stderr to .err file instead of test output, and tee it to stdout of the make process
Aaron Marcuse-Kubitza
06:29 PM Task #340 (Resolved): put together definitions of abstract BIEN terms
* specimen
* voucher
* taxonoccurrence
Aaron Marcuse-Kubitza
06:08 PM Revision 651: vegbien.sql: Updated name of UNIQUE constraint for specimen collectionnumber. Regenerated vegbien.ERD exports.
Aaron Marcuse-Kubitza
05:49 PM Revision 650: input.Makefile: Don't print "accept test" message when user aborted a test with Ctrl+C
Aaron Marcuse-Kubitza
05:34 PM Revision 649: inputs/SALVIAS/test: Accepted test outputs
Aaron Marcuse-Kubitza
05:33 PM Revision 648: input.Makefile: Also print message for accepting test output when diff fails
Aaron Marcuse-Kubitza
05:23 PM Revision 647: bin/map: Print a message when a database is successfully connected to
Aaron Marcuse-Kubitza
05:20 PM Revision 646: bin/map: Print a message when a database is successfully connected to
Aaron Marcuse-Kubitza
05:03 PM Revision 645: sql.py: Don't enclose PostgreSQL names in quotes because this disables case-insensitivity
Aaron Marcuse-Kubitza
04:58 PM Revision 644: sql.py: Use esc_name() to escape fields in SELECT statements
Aaron Marcuse-Kubitza
04:54 PM Revision 643: sql.py: Added esc_name() to escape identifiers like column names
Aaron Marcuse-Kubitza
04:27 PM Revision 642: vegbien.sql: Added comments to specimen.collectioncode_dwc and collectionnumber to differentiate them
Aaron Marcuse-Kubitza
04:22 PM Revision 641: vegbien.sql: Renamed authorspecimencode to collectionnumber to match its name in source data
Aaron Marcuse-Kubitza
04:18 PM Revision 640: input.Makefile: Use pipefail to cause a test to fail even when the output is filtered by grep. Print message for failing tests with command to run to accept the new test output.
Aaron Marcuse-Kubitza

01/24/2012

06:32 PM Task #339 (New): have "schema showdown"
Discuss different meanings everyone has for various ecological terms, especially:
* taxonoccurrence
* specimen
Aaron Marcuse-Kubitza
06:10 PM Revision 639: VegX-VegBIEN mapping: Map additional taxondetermination.determinationdate input formats straight through
Aaron Marcuse-Kubitza
06:08 PM Revision 638: test/input/SALVIAS_db.sh: Updated DB name
Aaron Marcuse-Kubitza
05:57 PM Revision 637: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
05:55 PM Revision 636: input.Makefile: Added documentation for why import errors for one input do not abort the import process for all inputs
Aaron Marcuse-Kubitza
05:53 PM Revision 635: input.Makefile: Determine DB name from input directory name, rather than DB file name
Aaron Marcuse-Kubitza
05:20 PM Revision 634: input.Makefile: Added documentation for accepting a test output
Aaron Marcuse-Kubitza
05:07 PM Revision 633: mappings/Makefile: Don't delete DwC-VegBIEN.specimens.csv in clean
Aaron Marcuse-Kubitza
05:05 PM Revision 632: VegBIEN: Renamed taxondetermination.*determination to is*
Aaron Marcuse-Kubitza
04:16 PM Revision 631: inputs/SALVIAS/test: Ignore test outputs
Aaron Marcuse-Kubitza
04:12 PM Revision 630: input.Makefile: Added test that generates VegBIEN.2-step.xml by mapping via a VegX.xml
Aaron Marcuse-Kubitza
03:39 PM Revision 629: input.Makefile: Added test that generates VegX.xml
Aaron Marcuse-Kubitza
03:29 PM Revision 628: input.Makefile: Added test that generates VegBIEN.xml
Aaron Marcuse-Kubitza
02:50 PM Revision 627: input.Makefile: Factored test/import.out out of DB section
Aaron Marcuse-Kubitza
02:43 PM Revision 626: input.Makefile: Renamed test/import.ref to import.out.ref. Changed syntax for accepting a test output to work with all types of test outputs.
Aaron Marcuse-Kubitza
02:32 PM Revision 625: Makefiles: Recurse into outermost subdir rather than bypassing it and going directly to innermost subdir
Aaron Marcuse-Kubitza
02:08 PM Revision 624: input.Makefile: Deal with inputs without a DB file, tests, verifications, etc.
Aaron Marcuse-Kubitza
02:07 PM Revision 623: inputs/Makefile: Don't use subdir makefiles because they are no longer needed
Aaron Marcuse-Kubitza
01:28 PM Revision 622: input.Makefile: Detect DB engine automatically from SQL file available in src subdir
Aaron Marcuse-Kubitza
01:00 PM Revision 621: input.Makefile: Factored as much as possible out of section for each DB engine
Aaron Marcuse-Kubitza
12:58 PM Revision 620: input.Makefile: Moved tests into test subdir
Aaron Marcuse-Kubitza
12:51 PM Revision 619: input.Makefile: Moved tests into test subdir
Aaron Marcuse-Kubitza
12:46 PM Revision 618: Added initial DwC-VegBIEN mappings spreadsheet with DwC terms
Aaron Marcuse-Kubitza
12:36 PM Revision 617: inputs/SALVIAS/verify: Updated to use new names for renamed tables
Aaron Marcuse-Kubitza
12:34 PM Revision 616: inputs/SALVIAS/verify: Updated to use new names for renamed tables
Aaron Marcuse-Kubitza
12:18 PM Revision 615: vegbien.sql: Removed no longer needed specimen.collector_id
Aaron Marcuse-Kubitza
12:16 PM Revision 614: VegX-VegBIEN mapping: Map collector name to new verbatimcollectorname field
Aaron Marcuse-Kubitza
12:11 PM Revision 613: vegbien.sql: Removed specimen.collectornumber_dwc and replaced it with verbatimcollectorname to reflect that the collectornumber_dwc is actually an ID of the specimen, and the collector's name is what we want to store
Aaron Marcuse-Kubitza

01/23/2012

05:33 PM Revision 612: mappings/Makefile: Run simplify_xpath on VegX-VegBIEN.organisms.csv
Aaron Marcuse-Kubitza
05:33 PM Revision 611: simplify_xpath: Be case sensitive to handle VegX correctly
Aaron Marcuse-Kubitza
05:18 PM Revision 610: VegX-VegBIEN mapping: Avoid using a dummy taxondetermination with role=collector
Aaron Marcuse-Kubitza
04:50 PM Revision 609: VegX-VegBIEN mapping: Map stem count to new stemcount field
Aaron Marcuse-Kubitza
04:48 PM Revision 608: VegX-VegBIEN mapping: Take advantage of aggregateoccurrence.count being optional
Aaron Marcuse-Kubitza
04:46 PM Revision 607: vegbien.sql: Made aggregateoccurrence.count to handle individuals data (for which count should be dynamically determined from # individual plants inside the aggregateoccurrence)
Aaron Marcuse-Kubitza
04:43 PM Revision 606: NYBG-VegBIEN mapping: Don't map dummy values to locationcode, etc. (e.g. in specimens data) because these tables are no longer required
Aaron Marcuse-Kubitza
04:42 PM Revision 605: vegbien.sql: Made several pointers to parent elements optional to deal with specimens data that might not have a location, etc.
Aaron Marcuse-Kubitza
04:29 PM Revision 604: vegbien.sql: Added taxondetermination UNIQUE constraint
Aaron Marcuse-Kubitza
04:18 PM Revision 603: VegX-VegBIEN mapping: Took advantage of location.confidentialitystatus being optional
Aaron Marcuse-Kubitza
04:16 PM Revision 602: VegX-VegBIEN mapping: Took advantage of userdefined.userdefinedtype being optional
Aaron Marcuse-Kubitza
04:13 PM Revision 601: vegbien.sql: Gave userdefined.userdefinedtype a default value
Aaron Marcuse-Kubitza
04:10 PM Revision 600: VegX-VegBIEN mappings: Took advantage of plantconcept.reference_id becoming optional
Aaron Marcuse-Kubitza
04:07 PM Revision 599: vegbien.sql: Made plantconcept.reference_id optional. Merge plantconcepts with no reference_id when eliminating duplicates.
Aaron Marcuse-Kubitza
04:05 PM Revision 598: PostgreSQL-MySQL.csv: Deal with all non-NOT NULL timestamp fields
Aaron Marcuse-Kubitza
04:00 PM Revision 597: vegbien.sql: Removed confusing plantconcept.plantname field since we are using plantname.plantname instead
Aaron Marcuse-Kubitza
03:54 PM Revision 596: VegBIEN: Renamed aux_role to role
Aaron Marcuse-Kubitza
03:47 PM Revision 595: VegX-VegBIEN mappings: Took advantage of several fields becoming optional
Aaron Marcuse-Kubitza
03:43 PM Revision 594: VegX-VegBIEN mappings: Took advantage of several fields becoming optional
Aaron Marcuse-Kubitza
03:36 PM Revision 593: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
03:36 PM Revision 592: vegbien.sql: taxonbinmethod points to stratumtype instead of stratum because stratumtype is a method table, but stratum is a measurements table. stratum does not point directly to stratummethod because it points to it via stratumtype.
Aaron Marcuse-Kubitza
03:04 PM Revision 591: vegbien.sql: Made taxondetermination.determinationdate optional because some determinations might not have a date
Aaron Marcuse-Kubitza
02:47 PM Revision 590: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
02:46 PM Revision 589: vegbien.sql: Added specimen.authorspecimencode
Aaron Marcuse-Kubitza
02:36 PM Revision 588: Adjusted vegbien.ERD.mwb
Aaron Marcuse-Kubitza
02:34 PM Revision 587: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
02:33 PM Revision 586: VegBIEN: Renamed sourceaccessionnumber to sourceaccessioncode to show that they are the data source's analog of accessioncode. Added sourceaccessioncode to all applicable tables because this is the database pkey, which is distinct from any author*code applied by the collector.
Aaron Marcuse-Kubitza
01:56 PM Revision 585: vegbien.sql: Changed taxonbinmethod_keys to UNIQUE INDEX to take advantage of COALESCE() for dealing with NULL values
Aaron Marcuse-Kubitza
01:43 PM Revision 584: vegbien.sql: Renamed taxonbin to taxonbinmethod to reflect that it does not contain actual organisms (those go in aggregateoccurrence), but rather defined a *method* of aggregating organisms
Aaron Marcuse-Kubitza
01:39 PM Revision 583: vegbien.sql: Removed taxonbin.count because that belongs in aggregateoccurrence and taxonbin is more similar to a sampling method. Added taxonbin UNIQUE constraint.
Aaron Marcuse-Kubitza
01:24 PM Revision 582: vegbien.sql: Do location duplicate elimination independently on code or lat/long, allowing duplicate entries with NULLs to exist when a location is incompletely specified
Aaron Marcuse-Kubitza
01:15 PM Revision 581: vegbien.sql: Require location to have either an authorlocationcode or a lat/long. Distinguish between regular and subplots in UNIQUE constraint.
Aaron Marcuse-Kubitza
01:10 PM Revision 580: vegbien.sql: Renamed location.latitude and longitude to publiclatitude, publiclongitude to reflect that they are not the actual lat/long. Switched to requiring reallatitude/reallongitude.
Aaron Marcuse-Kubitza
01:02 PM Revision 579: Added inputs/TurboVeg
Aaron Marcuse-Kubitza
12:58 PM Revision 578: vegbien.ERD.mwb: Deal with MySQL assuming that a timestamp field is NOT NULL
Aaron Marcuse-Kubitza
12:57 PM Revision 577: PostgreSQL-MySQL.csv: Deal with MySQL assuming that a timestamp field is NOT NULL
Aaron Marcuse-Kubitza
12:53 PM Revision 576: vegbien.sql: Made specimen.taxonoccurrence_id required
Aaron Marcuse-Kubitza
12:47 PM Revision 575: vegbien.sql: Made several fields optional, adding defaults where needed
Aaron Marcuse-Kubitza
12:47 PM Revision 574: PostgreSQL-MySQL.csv: Deal with PostgreSQL-style :: casts
Aaron Marcuse-Kubitza
12:14 PM Revision 573: NYBG mappings: Add mapping for CollectorNumber to specimen.collectornumber_dwc
Aaron Marcuse-Kubitza
12:08 PM Revision 572: vegbien.sql: Added specimen.collectornumber_dwc
Aaron Marcuse-Kubitza
12:03 PM Revision 571: VegBIEN: Renamed sourceid to author*code
Aaron Marcuse-Kubitza
12:01 PM Revision 570: VegBIEN: Renamed sourceid to author*code
Aaron Marcuse-Kubitza
11:44 AM Revision 569: mappings: Map ScientificNameAuthor to plantconcept with rank author
Aaron Marcuse-Kubitza
10:53 AM Revision 568: vegbien.sql: Removed sizeclass.*precision fields
Aaron Marcuse-Kubitza
10:50 AM Revision 567: vegbien.sql: Added count field to taxonBin
Aaron Marcuse-Kubitza
10:47 AM Revision 566: VegBIEN: Renamed place to locationplace
Aaron Marcuse-Kubitza
10:40 AM Revision 565: vegbien.sql.make: Exclude 'CREATE OR REPLACE PROCEDURAL LANGUAGE plpgsql' to avoid restore errors
Aaron Marcuse-Kubitza
10:20 AM Revision 564: repl: When matching words, treat _ as word boundary and ignore case.
Aaron Marcuse-Kubitza

01/20/2012

05:09 PM Revision 563: VegBIEN: Renamed individualplant.height to overallheight
Aaron Marcuse-Kubitza
03:02 PM Task #337 (Closed): VegBIEN table renames
E-mail from Brad on 2012-1-20:
Hi Mike,
You and Aaron have convinced me. Bob & I were advocating a return to th...
Aaron Marcuse-Kubitza
02:34 PM Revision 562: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
02:32 PM Revision 561: PostgreSQL-MySQL.csv: Convert double quotes to backticks (`)
Aaron Marcuse-Kubitza
02:10 PM Revision 560: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
02:09 PM Revision 559: vegbien.for_ERD.sql: Fixed syntax error in stratum table
Aaron Marcuse-Kubitza
02:00 PM Revision 558: Regenerated vegbien.ERD exports
Aaron Marcuse-Kubitza
01:39 PM Revision 557: vegbien.sql: Renamed sizeclass.*accuracy to *precision to reflect intended purpose
Aaron Marcuse-Kubitza
01:33 PM Revision 556: vegbien.sql: Added stem.diameteraccuracy
Aaron Marcuse-Kubitza
01:22 PM Revision 555: vegbien.sql: Removed taxonoccurrence.currentdetermination_id and originaldetermination_id because it's not possible to create a two-way pointer when the child-to-parent pointer is required (chicken-and-egg problem)
Aaron Marcuse-Kubitza
01:05 PM Revision 554: README.TXT: Added make empty_db
Aaron Marcuse-Kubitza
12:52 PM Revision 553: vegbien.sql: Added taxonoccurrence.currentdetermination_id and originaldetermination_id
Aaron Marcuse-Kubitza
12:41 PM Revision 552: vegbien.ERD.mwb: Added stratum and place tables
Aaron Marcuse-Kubitza
12:29 PM Task #338 (Resolved): put together samples of SALVIAS data that aggregates individuals and stems in different ways
* individual count
* stem count
* individuals
* stems
Aaron Marcuse-Kubitza

01/19/2012

06:06 PM Task #337: VegBIEN table renames
"observation" can be confusing, because it's not obvious what it's an observation of (a plot, an organism, a stem?). ... Aaron Marcuse-Kubitza
06:02 PM Task #337 (Closed): VegBIEN table renames
E-mail from Brad:
Conceptual definitions of tables:
observation: permanent attributes of a unit or area of obse...
Aaron Marcuse-Kubitza
05:32 PM Task #336 (Resolved): create VegBIEN ERD
Now live on [[VegBIEN schema]]! Aaron Marcuse-Kubitza
04:56 PM Task #336: create VegBIEN ERD
I discovered that MySQL Workbench can synchronize its model with an existing CREATE script. So, it looks like we can ... Aaron Marcuse-Kubitza
04:54 PM Task #336 (Resolved): create VegBIEN ERD
Aaron Marcuse-Kubitza
05:28 PM Revision 551: Added vegbien.ERD.png
Aaron Marcuse-Kubitza
05:24 PM Revision 550: Added vegbien.ERD.svg
Aaron Marcuse-Kubitza
05:21 PM Revision 549: Added vegbien.ERD.pdf
Aaron Marcuse-Kubitza
05:19 PM Revision 548: Added vegbien ERD
Aaron Marcuse-Kubitza
05:19 PM Revision 547: Ignore *.bak in schemas
Aaron Marcuse-Kubitza
05:07 PM Revision 546: VegBIEN: specimen links directly to taxonoccurrence rather than indirectly via individualplant
Aaron Marcuse-Kubitza
04:43 PM Revision 545: vegbien.sql: Made voucher fields NOT NULL
Aaron Marcuse-Kubitza
04:41 PM Revision 544: vegbien.sql: Added voucher table
Aaron Marcuse-Kubitza
04:26 PM Revision 543: VegBIEN: specimen links directly to taxonoccurrence rather than indirectly via individualplant
Aaron Marcuse-Kubitza
04:22 PM Revision 542: VegBIEN: specimen links directly to taxonoccurrence rather than indirectly via individualplant
Aaron Marcuse-Kubitza
04:12 PM Revision 541: Create schemas/vegbien.for_wiki.sql from schemas/vegbien.for_ERD.sql
Aaron Marcuse-Kubitza
04:11 PM Revision 540: Added lib/ERD-wiki.csv to convert ERD SQL to human-readable wiki format
Aaron Marcuse-Kubitza
04:10 PM Revision 539: schemas/vegbien.for_ERD.sql: Removed VegBank-internal fields from taxonoccurrence
Aaron Marcuse-Kubitza
03:51 PM Revision 538: Removed no longer needed schemas/vegbien.my.sql
Aaron Marcuse-Kubitza
03:51 PM Revision 537: PostgreSQL-MySQL.csv: Use \g<0> instead of \0 in replacements because \0 not supported. Replace fulltext with fulltext__ (note fulltext_ doesn't work).
Aaron Marcuse-Kubitza
01:34 PM Revision 536: Added schemas/vegbien.for_ERD.sql and generated MySQL
Aaron Marcuse-Kubitza
01:34 PM Revision 535: Regenerated schemas/vegbien.for_ERD.sql
Aaron Marcuse-Kubitza
01:33 PM Revision 534: PostgreSQL-MySQL.csv: Deal with serial NOT NULL fields
Aaron Marcuse-Kubitza
01:22 PM Revision 533: Generate MySQL DDL schemas/vegbien.my.sql from vegbien.sql
Aaron Marcuse-Kubitza
01:22 PM Revision 532: Added repl spreadsheet to map PostgreSQL DDL to MySQL
Aaron Marcuse-Kubitza
12:56 PM Revision 531: Ignore src subdirectories in inputs
Aaron Marcuse-Kubitza
12:46 PM Revision 530: VegBIEN: Use stem instead of individualplant to store diameter/dbh
Aaron Marcuse-Kubitza
11:35 AM Task #302 (Resolved): Make changes to VegBIEN schema
Extensive changes have been made Aaron Marcuse-Kubitza
11:33 AM Task #331 (Resolved): use VegX's individualOrganismObservation/relatedItem to model stems
Stems are now modeled in VegX. (Note that _VegBIEN_ stems import not complete yet) Aaron Marcuse-Kubitza
11:32 AM Task #319: Update statistics/lists of user-defined fields in use in VegX and VegBIEN
I've done this for NYBG to VegX Aaron Marcuse-Kubitza
11:31 AM Task #322 (Resolved): get access to raw CTFS data
I have the data from Shash, including a SQL dump Aaron Marcuse-Kubitza
11:30 AM Task #334 (Resolved): which DarwinCore attributes do we want to include in the specimen table?
Desired fields have been added or moved from other tables Aaron Marcuse-Kubitza
11:30 AM Task #314: Import CTFS data
I have data from Shash and am working with her to make sure we can parse it Aaron Marcuse-Kubitza
11:29 AM Task #318 (Resolved): decide whether to switch to VegX 1.5.2
In fact, we will switch to 1.5.3, which is now available Aaron Marcuse-Kubitza

01/18/2012

05:57 PM Revision 529: vegbien.sql: Removed no longer needed taxondetermination.plantname_id
Aaron Marcuse-Kubitza
05:53 PM Revision 528: VegBIEN: Made taxondetermination.plantconcept_id required and plantname_id optional
Aaron Marcuse-Kubitza
04:28 PM Revision 527: vegbien.sql: Added specimen.taxonoccurrence_id fkey
Aaron Marcuse-Kubitza
03:47 PM Revision 526: vegbien.sql: Removed no longer needed taxondetermination.individualplant_id, specimen_id fields
Aaron Marcuse-Kubitza
01:57 PM Revision 525: VegBIEN: Renamed plot to location to reflect that a location can also be used for specimens
Aaron Marcuse-Kubitza
01:29 PM Revision 524: vegbien.sql: Added individualplant.stemcount
Aaron Marcuse-Kubitza
12:38 PM Revision 523: vegbien.sql: Added accessioncode fields to existing, modified tables
Aaron Marcuse-Kubitza
12:21 PM Revision 522: vegbien.sql: Added accessioncode fields to new tables
Aaron Marcuse-Kubitza

01/17/2012

04:56 PM Revision 521: vegbien.sql: Removed no longer needed individualplant.taxonarea
Aaron Marcuse-Kubitza
04:40 PM Revision 520: VegBIEN: Moved taxondetermination.sourceaccessionnumber and museum_id to specimen
Aaron Marcuse-Kubitza
03:32 PM Revision 519: vegbien.sql: Split taxondetermination UNIQUE CONSTRAINT into one constraint for each main key (taxonoccurrence_id, specimen_id, individualplant_id) to deal with Postgres treating a row with NULL in any of those columns as distinct. Added CHECK constraint to require one of taxonoccurrence_id, specimen_id, individualplant_id to be NOT NULL.
Aaron Marcuse-Kubitza
02:47 PM Revision 518: VegBIEN: Renamed museumaccessionnumber to sourceaccessionnumber
Aaron Marcuse-Kubitza
02:42 PM Revision 517: VegBIEN: Renamed authorcode to sourceid
Aaron Marcuse-Kubitza
02:33 PM Revision 516: vegbien.sql: Updated sizeclass to use ranges for parameters and remove taxonarea
Aaron Marcuse-Kubitza
 

Also available in: Atom