Activity
From 01/27/2012 to 02/25/2012
02/25/2012
- 08:16 PM Revision 1149: vegbien.sql: Added plantname unique constraint
- 08:01 PM Revision 1148: bin/map: Use new util.ListDict and util.WrapIter to simplify getting rows by column name instead of index, and to enable a row to be printed with its column names in error messages
- 08:00 PM Revision 1147: util.py: Added WrapIter to wrap an iterator and ListDict to view a list as a dict
- 07:38 PM Revision 1146: bin/map: Use new util.list_flip()
- 07:37 PM Revision 1145: util.py: Added list_flip()
- 07:02 PM Revision 1144: env_password: Fixed to set the environment variable in the calling shell. Do this by cc-ing the tty only on messages before the "Enter password" prompt, because the redirect creates a subshell which causes the env var to only be set within that subshell.
- 06:18 PM Revision 1143: inputs/NYBG-CSV/maps/DwC.specimens.csv: Removed mappings that are already present in mappings/DwC1-DwC2.specimens.csv. This map now contains only the mappings where NYBG-CSV differs from standard DwC1.
- 06:14 PM Revision 1142: inputs/NYBG/maps/DwC.specimens.csv: Removed mappings that are already present in mappings/DwC1-DwC2.specimens.csv. This map now contains only the mappings where NYBG differs from standard DwC1.
- 05:58 PM Revision 1141: Remove accidentally-committed temp file inputs/NYBG/DwC.specimens2.csv
- 05:56 PM Revision 1140: mappings/Makefile: Generate DwC.self.specimens.csv from DwC-VegBIEN.specimens.csv for use in creating full via maps for inputs
- 05:40 PM Revision 1139: input.Makefile: Generate full via maps from input via maps by appending mappings from the via format to itself when available
- 04:30 PM Revision 1138: inputs/NYBG/maps/DwC.specimens.csv: Changed label to "NYBG-DwC" to take advantage of automatic filling in of DwC mappings not specified in the NYBG map
- 04:28 PM Revision 1137: subtract: Support custom column numbers to compare on (instead of just input col). Added ignore option to continue even if input columns don't match.
- 04:26 PM Revision 1136: bin/map: DB inputs: Get all rows in one query (hopefully a significant optimization). Allow maps to contain entries for columns that are not in the DB table.
- 04:22 PM Revision 1135: sql.py: select(): Select all fields if fields == None. Replaced col(cur, idx) with col_names(cur) because an iterator is easier to use than getting by index.
- 03:57 PM Revision 1134: bin/map: Fixed bug in previous implementation of allowing maps for CSV inputs to contain entries for columns that are not in the CSV file
- 03:45 PM Revision 1133: bin/map: Allow maps for CSV inputs to contain entries for columns that are not in the CSV file
- 02:54 PM Revision 1132: Use new sort_map instead of manually specifying the sort order
- 02:54 PM Revision 1131: Added sort_map to sort a map spreadsheet in the standard order
- 02:43 PM Revision 1130: Removed no longer needed join_passthru, because join_union_sort now serves its purpose
- 02:42 PM Revision 1129: Don't generate mappings/for_review/DwC-VegBIEN.specimens.csv because it's a derived map with lots of duplicated mappings for the various DwC versions
- 02:41 PM Revision 1128: mappings/Makefile: Generate DwC-VegBIEN.specimens.csv directly from DwC1-DwC2 and DwC2-VegBIEN mappings by using join_union_sort with header_num=1, rather than via intermediate DwC1-VegBIEN.specimens.csv
- 02:37 PM Revision 1127: union: Added header_num option to select which map's header to use as the output header
- 02:28 PM Revision 1126: Rename join_sort to join_union_sort and have it run union in ignore mode. This will automatically append the joined map when the input map is a derivative of the joined map, such as for NYBG-DwC.
- 02:25 PM Revision 1125: union: Pass through map 0, so that if ignore is set, the input map will still be output. Allow either map's input label to contain the other's input label to enable e.g. appending mappings for an older input version to those for a newer input version.
- 01:43 PM Revision 1124: DwC1-DwC2 mapping: Changed input label to DwC1, which is allowed by the now relaxed label constraints imposed by union
- 01:42 PM Revision 1123: union: Check if two maps can be combined based on whether map 0 column 0 label *contains* map 1 column 0 label instead of being equal. This allows map 0's input 0 root to contain the datasource name as well as a format that allows it to be combined with a more general map. Added ignore flag to not print an error if column labels don't match.
- 01:39 PM Revision 1122: bin/map: Support optional data format tag in map spreadsheet labels, used by union to check if two maps can be combined
- 01:01 PM Revision 1121: mappings: Added DwC1-DwC2.specimens.csv to core maps so it gets cleaned up
- 12:57 PM Revision 1120: Only generate for_review mappings of core maps and end products
- 12:56 PM Revision 1119: Generate DwC-VegBIEN mapping as union of DwC1 and DwC2 mappings
02/24/2012
- 08:00 PM Revision 1118: Generate DwC-VegBIEN mapping as union of DwC1 and DwC2 mappings
- 07:40 PM Revision 1117: NYBG DB mapping: Removed IdentifiedDate and CollectedDate mappings because they are generated from the year/month/day
- 07:39 PM Revision 1116: Added mappings/for_review/DwC1-VegBIEN.specimens.csv
- 07:35 PM Revision 1115: Added DwC1-DwC mapping. Generate DwC1-VegBIEN mapping automatically.
- 07:11 PM Revision 1114: Regenerated vegbien.ERD exports
- 07:08 PM Revision 1113: vegbien.sql: Renamed _keys unique constraints/unique indexes to _unique to better reflect their purpose
- 06:54 PM Revision 1112: vegbien.sql: Added method.diameterheight to store DBH height
- 06:44 PM Revision 1111: VegBIEN: Moved plantstatus.plantlevel to plantname.rank because the taxonomic rank is a property of the name itself
- 06:43 PM Revision 1110: PostgreSQL-MySQL.csv: Fixed custom types translation to match shorter type names
- 06:09 PM Revision 1109: vegbien.sql: Added plantstatus unique constraint
- 06:07 PM Revision 1108: DwC-VegBIEN mapping: Map datasource name via DwC institutionCode
- 05:42 PM Revision 1107: Regenerated vegbien.ERD exports
- 05:40 PM Revision 1106: vegbien.ERD.mwb: Lined up logo and legend with other ERD elements
- 05:35 PM Revision 1105: vegbien.sql: Renamed methodgrowthform.growthformmethod_id to submethod_id. Added methodtaxonclass.submethod_id (similar to methodgrowthform.submethod_id).
- 05:27 PM Revision 1104: vegbien.sql: Added methodgrowthform.growthformmethod_id for specifying a method used by just the growthform
- 05:14 PM Revision 1103: vegbien.ERD.mwb: Rearranged legend to more closely match layout of ERD
- 04:51 PM Revision 1102: vegbien.sql: Reordered plantstatus fields to put the most important fields at the top, which will be visible in the ERD
- 04:42 PM Revision 1101: vegbien.sql: Replaced method.taxonclassincluded,taxonclassexcluded with new many:many methodtaxonclass table. Added methodgrowthform, growthform tables to do the same thing as methodtaxonclass for growth forms.
- 03:53 PM Revision 1100: vegbien.sql: method: Added comment on reference_id
- 03:44 PM Revision 1099: VegBIEN: Moved plotmethod fields to method because they can also apply to strata. Removed no longer used plotmethod table.
- 03:13 PM Revision 1098: input.Makefile: input DB creation: Removed "IF NOT EXISTS" because that check is handled by $(dbExists)
- 03:02 PM Revision 1097: input.Makefile: Don't try to recreate an input DB if it already exists
- 03:01 PM Revision 1096: Added UArizona DB input
- 02:42 PM Revision 1095: Renaming UArizona to UArizona-CSV because there is also a DB input in bien2_staging.ariz_raw on nimoy
- 02:31 PM Revision 1094: Added UArizona input
- 12:46 PM Revision 1093: env_password: Fixed bug where exit command would not cause it to exit, because pipefail shell option was not set. Moved automatic exiting of the calling script into env_password itself.
- 12:26 PM Revision 1092: map: Exit if password not set
- 12:18 PM Revision 1091: env_password: cc stderr if it's a log file
02/23/2012
- 06:49 PM Revision 1090: env_password: Print all messages to /dev/tty so the user sees them even if stderr is redirected to a log file. Exit if password not already set, because e.g. scripts run in the background will not be able to prompt for it.
- 05:32 PM Revision 1089: input.Makefile: Don't have make import call verify, because the user often runs import as a test and will not want the output cluttered with verification information. Also, the full imports for which this was intended are often run asynchronously, so that the user will not see the output anyway.
- 05:28 PM Revision 1088: input.Makefile: Don't abort on verification errors, which are expected during development
02/21/2012
- 06:21 PM Revision 1087: SALVIAS tests: Fixed invalid accepted test outputs due to not running `make empty_db` before running tests when using the no-redo optimization shortcut
- 06:14 PM Revision 1086: SALVIAS mappings: Fixed plot key mappings to map the correct values to subplot and parent plot
- 05:36 PM Revision 1085: vegbien.sql: locationevent: Added unique constraint for subplots based on subplot location
- 05:02 PM Revision 1084: SALVIAS-db VegX mapping: Map subplots correctly the way SALVIAS-CSV does
- 04:54 PM Revision 1083: SALVIAS verification: Updated to schema changes
- 04:42 PM Revision 1082: input.Makefile: Fixed syntax error in verify %.ref target (outdated variable name)
- 04:33 PM Revision 1081: input.Makefile: Halt psql commands on first error
- 04:27 PM Revision 1080: vegbien.sql: Removed location.authorlocationcode because it's now stored in locationevent as an author-specific setting
- 04:24 PM Revision 1079: vegbien.sql: locationevent: Redid unique constraints to avoid applying authorlocationcode-only duplicate elimination to subplots
- 04:16 PM Revision 1078: SALVIAS mappings: Map SiteCode/plot_code to locationevent.authorlocationcode because locationevent is now the place to store author-specific plot information
- 04:10 PM Revision 1077: SALVIAS mappings: Fixed PlotID mapping to go to locationevent.sourceaccessioncode
- 04:06 PM Revision 1076: VegBIEN: Renamed locationevent.authoreventcode to authorlocationcode to reflect that datasources usually use an author-defined code for a plot rather than a plot event
- 04:03 PM Revision 1075: vegbien.sql: locationevent: Redid unique constraints to handle datasources that treat the authoreventcode as an authorlocationcode. Eventually, authoreventcode will be renamed to authorlocationcode.
- 03:51 PM Revision 1074: vegbien.sql: locationevent: Redid unique constraints to work properly for all fully-specified combinations of keys
- 03:31 PM Revision 1073: VegBIEN mappings: Mapped datasource name to new project.datasource. Fixes project duplicate elimination.
- 03:16 PM Revision 1072: vegbien.sql: Renamed project.reference_id to datasource_id and pointed it to party, to match locationevent, etc.
- 03:02 PM Revision 1071: VegBIEN mappings: Mapped current lat/long to centerlat/long as well so location duplicate elimination will work properly
- 03:01 PM Revision 1070: xpath.py: Added support for common subpath after split path's {}
- 01:30 PM Revision 1069: sql.py: put(): When encountering a DuplicateKeyException, use dict_subset_right_join to fill in explicit NULL values for columns which don't have data. This causes the database to use the UNIQUE constraint's index to look up the record, instead of relying on individual column indexes for the columns that did have data, which may or may not be available.
- 01:27 PM Revision 1068: util.py: Added DefaultDict to wrap collections.defaultdict with a simple value passed in the constructor, defaulting to None. Added dict_subset_right_join() to fill in None for subset keys that don't exist.
- 01:06 PM Revision 1067: vegbien.sql: Added method and plotmethod UNIQUE indexes
- 01:04 PM Revision 1066: vegbien.ERD.mwb: Removed embargo table from ERD because its functionality is provided in location.confidentialitystatus,confidentialityreason
- 12:36 PM Revision 1065: Regenerated vegbien.ERD exports
- 12:34 PM Revision 1064: vegbien.sql: Moved locationevent method fields to plotmethod and method. Added comments to method/plotmethod fields, as provided by Michael Lee.
- 12:15 PM Revision 1063: VegX-VegBIEN mapping: Mapped locationevent.methodnarrative to new plotmethod table
02/18/2012
- 07:18 PM Revision 1062: Regenerated vegbien.ERD exports
- 07:15 PM Revision 1061: VegX-VegBIEN mapping: Map sourceaccessioncode and voucher (catalognumber_dwc) to correct place. SALVIAS mappings: Map SourceVoucher as an alternative to coll_number.
- 06:46 PM Revision 1060: vegbien.sql: Removed VegBank-internal tables (including user account tables) because they do not belong in the ecological database. Any web interface should store user account information, cached queries, etc. in a separate interface-specific database.
- 06:34 PM Revision 1059: VegX-VegBIEN mapping: Mapped stem tags to new stemtag table
- 05:47 PM Revision 1058: vegbien.sql: Renamed planttag to stemtag and made it a child of stemobservation. Removed trait table from ERD because it's not used for the purpose we want to use traits for.
- 05:05 PM Revision 1057: vegbien.sql: Removed no longer used location.reference_id. Datasource scoping is now done on locationevent instead, so that locations can be shared across datasources that refer to the same plot or point.
- 04:56 PM Revision 1056: VegX-VegBIEN mapping: Map datasource name (/_ignore/inLabel) to new locationevent.datasource instead of location.reference
- 04:45 PM Revision 1055: vegbien.sql: Added locationevent.datasource_id
- 04:33 PM Revision 1054: vegbien.sql: locationevent: Removed VegBank-internal interp_* fields
- 04:28 PM Revision 1053: VegBIEN: Renamed specimenreplicate.reference_id to datasource_id and pointed it to party instead of reference, since party is better optimized for storing names
- 04:14 PM Revision 1052: DwC mappings: Mapped datasource name to specimenreplicate.reference instead of location.reference
- 04:08 PM Revision 1051: DwC mappings: Mapped specimen description via fieldNotes instead of custom bien.specimenDescription field
- 03:52 PM Revision 1050: VegBIEN: Moved specimenreplicate.verbatimcollectorname to taxonoccurrence since it can also apply to aggregateoccurrences. Removed no longer needed taxonoccurrence fields which are now in taxondetermination.
- 03:31 PM Revision 1049: SALVIAS mappings: Mapped habit to growthForm (user-defined field) instead of habit
- 03:24 PM Revision 1048: DwC-VegBIEN mapping: Convert latitude/longitude values of exactly zero to NULL
- 03:23 PM Revision 1047: xml_func.py: Added _nullIf
- 03:22 PM Revision 1046: util.py: Fixed cast() to not cast a subclass to a superclass (which doesn't make sense in a dynamically-typed language). Added none_if().
- 02:50 PM Revision 1045: util.py: Removed locale import since it's no longer used by util
- 02:31 PM Revision 1044: NYBG-DwC mappings: Map Vegetation to habitat (merged with Habitat). DwC-VegBIEN mapping: Removed remaining mappings to plantobservation.
- 02:22 PM Revision 1043: DwC-VegBIEN mapping: Added datasource name to location.reference using /_ignore/inLabel
- 02:00 PM Revision 1042: profiling.py: Support Python before 2.7 by using new dates.total_seconds(). Also use dates.now() to ensure datetimes always have a timezone.
- 01:59 PM Revision 1041: dates.py: Fixed timestamp() to deal with microseconds correctly by adding them after time.mktime()
- 01:41 PM Revision 1040: dates.py: Deal properly with different timezones by using external dateutil package. Added total_seconds() to replace datetime.timedelta.total_seconds() on Python before 2.7.
- 12:41 PM Revision 1039: vegbien.sql: *method tables: Added table comments
02/17/2012
- 06:08 PM Revision 1038: VegX-VegBIEN mapping: Reordered 2-step-only mappings that use /_ignore/inLabel so they run at the same time as other mappings that set the field that uses /_ignore/inLabel. This fixes almost all of the failing 2-step tests.
- 05:53 PM Revision 1037: Regenerated vegbien.ERD exports
- 05:46 PM Revision 1036: vegbien.sql: method: Added lengthunits field
- 05:36 PM Revision 1035: vegbien.sql: Changed types of numerical plotmethod fields to double precision
- 05:32 PM Revision 1034: vegbien.sql: method, plotmethod: Added comments to fields
- 05:27 PM Revision 1033: vegbien.ERD.mwb: Adjusted lines
- 05:24 PM Revision 1032: vegbien.sql: Added plotmethod. locationevent points to plotmethod instead of directly to method
- 04:47 PM Revision 1031: vegbien.sql: Point to covermethod from method instead of locationevent
- 04:39 PM Revision 1030: vegbien.sql: Removed no longer needed sizeclass table (whose fields are now in method)
- 04:32 PM Revision 1029: vegbien.sql: Replaced stratumtype, stratummethod with method
- 04:25 PM Revision 1028: vegbien.sql: Attach method to aggregateoccurrence instead of taxonoccurrence
- 04:20 PM Revision 1027: vegbien.sql: Removed methodtrait* tables and added first-class method attributes as first-class fields of method. Removed *method tables from the ERD that will be replaced by method.
- 04:04 PM Revision 1026: vegbien.sql: Removed location.dsgpoly because it is now locationdetermination.footprintgeometry_dwc
- 03:59 PM Revision 1025: VegBIEN mappings: Remap to new locationdetermination fields
- 03:44 PM Revision 1024: VegBIEN: Renamed location.reallatitude,reallongitude to centerlatitude,centerlongitude to reflect that it's now a value calculated from the centroid of the current locationdetermination
- 03:35 PM Revision 1023: vegbien.sql: locationdetermination: Reordered fields
- 03:32 PM Revision 1022: vegbien.sql: locationdetermination.coordsaccuracy: Added comment with units
- 03:29 PM Revision 1021: vegbien.sql: locationdetermination: Added determination status columns from taxondetermination
- 03:20 PM Revision 1020: vegbien.sql: locationdetermination: Added coordinates-related fields
- 03:08 PM Revision 1019: VegX-VegBIEN mapping: Include the datasource name (now provided by map in /_ignore/inLabel) in the appropriate places in both VegX and VegBIEN
- 02:47 PM Revision 1018: bin/map: Removed metadata values feature since the syntax used was causing problems with mappings starting with a ":", and metadata can instead be stored as attributes of the primary key's mapping
02/16/2012
- 06:07 PM Revision 1017: xml_dom.py: Fixed bug in parent() where it didn't account for NodeParentIter's first element returned being the current node, not its parent. Refactored parent() to use parentNode directly, and NodeParentIter to use parent(), instead of the other way around.
- 05:38 PM Revision 1016: xml_dom.py: Fixed bug in parent() where incorrect variable name was used
- 05:05 PM Revision 1015: VegX-VegBIEN mapping: Use the input data source's label (e.g. SALVIAS) everywhere a reference is needed
- 05:04 PM Revision 1014: bin/map: Store the input data source's label (e.g. SALVIAS) in the output XML tree for use by references in the mappings
02/14/2012
- 06:57 PM Revision 1013: xpath.py: get(): Fixed bug where it would try to create a node named . or .. if . or .. didn't have matching attributes. Now it will just reuse the current or parent node, but create any needed attrs if create is True.
- 06:52 PM Revision 1012: util.py: Added list_eq_is() to compare two lists using is
- 06:23 PM Revision 1011: xpath.py: Don't allow rooted attributes (doesn't make sense), in case someone tries to do elem[/rooted_attr]
- 05:52 PM Revision 1010: bin/map: Moved root.clear() into separate function prep_root() that can be called whenever needed
- 05:43 PM Revision 1009: xpath.py: Added get() support for references (different from pointers) to dynamically set the value of an attribute
- 05:21 PM Revision 1008: util.py: Added list_get()
- 05:10 PM Revision 1007: util.py: Added is_list()
- 05:00 PM Revision 1006: bin/map: Use var doc0_root for quick reference to doc0's root
- 04:57 PM Revision 1005: xpath.py: get(): Go to root when empty element is encountered at the beginning of an XPath. Added allow_rooted parameter to turn off this functionality when XPaths with a leading slash should not be considered rooted.
- 04:32 PM Revision 1004: xpath.py: Don't consider a path starting with "." to be rooted. Do this by not automatically translating an empty path name to ".".
- 04:18 PM Revision 1003: xpath.py: Added is_rooted()
- 04:16 PM Revision 1002: xpath.py: Added elem_is_empty()
- 04:10 PM Revision 1001: xpath.py: Added documentation labels to each section
- 04:04 PM Revision 1000: xpath.py: Added support for getting the parent node when encountering ".."
- 04:02 PM Revision 999: xml_dom.py: Added parent() to get parent node without recursing past the root node to the document object. Documented that NodeParentIter incorporates this sanity check.
- 03:44 PM Revision 998: xpath.py: get(): Renamed parent to root to better reflect that it's the starting point for the search. Calling it parent will later be confusing when we want to get the parent node using "..".
- 03:31 PM Revision 997: xpath.py: Added parser support for attribute values that are references to another part of the XML tree
- 03:21 PM Revision 996: xml_func.py: Fixed module description comment to reflect that not all XML funcs generate text
- 03:20 PM Revision 995: xml_func.py: Refactored to add funcs to the module funcs variable as they are defined. Renamed defined functions to the name of the corresponding XML function.
- 03:01 PM Revision 994: xml_func.py: Added _ignore func to "comment out" an XML subtree
- 02:55 PM Revision 993: input.Makefile: Fixed error message when no DB file found so that it doesn't incorrectly imply that PostgreSQL inputs are supported
- 02:54 PM Revision 992: input.Makefile: Fixed error message when no DB file found so that it doesn't incorrectly imply that PostgreSQL inputs are supported
- 02:42 PM Revision 991: input.Makefile: Don't run tests in verbose mode because the run time stats, etc. are not relevant
- 02:41 PM Revision 990: bin/map: Only print error/run time stats in verbose mode. input.Makefile: Run import in verbose mode so that error/run time stats are still printed.
- 02:36 PM Revision 989: Moved value to string conversion functions infrom util.py to new module format.py
- 02:32 PM Revision 988: exc.py, profiling.py: Use util.int2str() to print # iters with thousands separators
- 02:32 PM Revision 987: util.py: Added int2str()
- 02:26 PM Revision 986: bin/map: Document that the exit status is the # of errors in the import, up to the maximum exit status
- 02:24 PM Revision 985: exc.py: Generalize ExTracker to not just print the # of errors at exit. Instead, provide an exit() method that the ExTracker creator can call at exit to set the exit status to the # of errors. This fixes the Python bug where a benign error message was printed if SystemExit was raised in an atexit function.
- 02:15 PM Revision 984: bin/map: Set ExPercentTracker's iter_text. Start ExPercentTracker after input processing, because errors in command line options should just end the program and don't need to be tracked.
- 02:13 PM Revision 983: exc.py: ExPercentTracker: Added ability to set custom iter_text, similar to ItersProfiler
- 02:07 PM Revision 982: bin/map: Use profiling.ItersProfiler. Refactored input row count calculation to have each function aggregate and return the row count, and then display the row count and statistics that depend on it at the end of the program.
- 02:06 PM Revision 981: Added profiling.py to time operations and provide the user with statistical information
- 02:03 PM Revision 980: util.py: Added basic to_si() to add SI prefix to value
- 01:02 PM Revision 979: util.py: Added format_str() to use locale-specific formatting settings, including thousands separator. Use it in to_percent().
- 12:48 PM Revision 978: bin/map: Use new ExPercentTracker to print error rate (% of # rows) when program exits
- 12:47 PM Revision 977: exc.py: Added ExPercentTracker to track errors as % of iterations
- 12:47 PM Revision 976: util.py: Added to_percent()
- 12:07 PM Revision 975
- 12:02 PM Revision 974: exc.py: print_ex(): Declare emph param as a keywork param instead of popping it from **format
- 11:54 AM Revision 973: exc.py: print_ex(): Declare emph param as a keywork param instead of popping it from **format
02/13/2012
- 04:38 PM Revision 972: inputs/SALVIAS/maps/VegX.organisms.csv: Mapped OrigSpecies and OrigGenus combined to new plantlevel Binomial
- 04:35 PM Revision 971: xpath.py: Fixed bug where value of XPath (used for copying to other branches) is retrieved after first XPath element is popped rather than before, which can sometimes leave an empty XPath for value() to run on
- 03:33 PM Revision 970: mappings/DwC-VegBIEN.specimens.csv: Fixed bien.vegetation mapping to point to commconcept->commname. Fixed bien.substrate mapping to point to locationevent.landscapenarrative.
- 03:25 PM Revision 969: inputs/NYBG/maps/DwC.specimens.csv: Mapped CoordinatePrecision using _noCV
- 03:24 PM Revision 968: xml_func.py: Added _noCV func to check that non-ratio-scale data does not contain CV values
- 02:34 PM Revision 967: mappings/DwC-VegBIEN.specimens.csv: Fixed locality fields mapping to go to location.locationnarrative
- 02:05 PM Revision 966: input.Makefile: For all input types, including DB, import each table in a separate map invocation
- 01:58 PM Revision 965: xml_func.py: _range: Treat a None from or to value as an unknown (a la SQL NULL) and return None instead of raising a SyntaxException
- 01:57 PM Revision 964: xml_dom.py: NodeTextEntryIter: Convert empty entries (including entries containing error comments) to None
- 01:48 PM Revision 963: xml_dom.py: replace(): Added support for new node that's None (deletes existing node)
- 12:59 PM Revision 962: xml_func.py: Put SyntaxException's cause on same line as error message so that the whole error is treated as distinct by error_stats
- 12:47 PM Revision 961: Added errors_filter_before and errors_filter_after to prepare `map` error messages for easy filtering and then restore line breaks
- 12:46 PM Revision 960: error_stats: Fixed to work on Mac
- 11:26 AM Revision 959: error_stats: Simplified to use uniq --count option
- 11:21 AM Revision 958: input.Makefile: Print error message if no input file found (for file input type). This fixes a bug where map would just take input from stdin when no input file redirect or input DB env vars were specified.
- 11:10 AM Revision 957: map: Map standard DB names to original DB names on nimoy
- 10:29 AM Revision 956: Regenerated vegbien.ERD exports
- 10:26 AM Revision 955: vegbien.sql: Added methodtrait and methodtraitname tables
- 10:24 AM Revision 954: PostgreSQL-MySQL.csv: Handle array types
- 09:54 AM Revision 953: vegbien.ERD.mwb: Recolored plant tables to all have the same color, distinct from the occurrence color
- 09:32 AM Revision 952: mappings/VegX-VegBIEN.organisms.csv: Added mappings for SALVIAS fields with no join mapping. This fixes the last of the "no join mapping" errors.
- 09:27 AM Revision 951: input.Makefile: svn_props: Set svn:ignore on maps subdirs
- 09:26 AM Revision 950: inputs/SALVIAS-CSV/maps/VegX.plots.csv: Fixed mappings without a join mapping in VegX-VegBIEN.*.csv
- 09:20 AM Revision 949: VegX mappings: Gentry DBH mapping: Use VegX's attribute and method tables
- 09:05 AM Revision 948: mappings/VegX-VegBIEN.organisms.csv: Removed no longer used mapping to taxondetermination.determinationdate. This also prevents ever creating a taxondetermination without a plantconcept.
02/11/2012
- 07:31 PM Revision 947: bin/map: Added redo option to control whether the database is emptied before inserting new data. Can be used to turn off emptying the DB in test mode, because this is often slow and is not needed if you are running tests on an empty testing database.
- 07:25 PM Revision 946: opts.py: env_flag(): Added support for default value if unset
- 07:25 PM Revision 945: opts.py: env_flag(): Added support for default value if unset
- 07:16 PM Revision 944: bin/map: Use env_flag()'s new env_names usage support to print flags usage
- 07:16 PM Revision 943: opts.py: Added env_names usage support to env_flag()
- 07:05 PM Revision 942: mappings/VegX-VegBIEN.organisms.csv: Removed no longer needed mapping for taxonDetermination/note
- 07:03 PM Revision 941: inputs/SALVIAS-CSV/maps/VegX.organisms.csv: Map cfaff to taxonConcept/fit, which maps to taxondetermination.taxonFit
- 06:59 PM Revision 940: inputs/SALVIAS/maps/VegX.organisms.csv: Map cfaff to taxonConcept/fit, which maps to taxondetermination.taxonFit
- 06:39 PM Revision 939: join: Print a warning if no join mapping found (in addition to adding this warning to the comments column)
- 06:27 PM Revision 938: Removed no longer needed inputs/NYBG/maps/VegX.organisms.csv because NYBG is now mapped via DwC
- 06:26 PM Revision 937: mappings/VegX-VegBIEN.organisms.csv: Removed mappings used only by NYBG, because NYBG now maps via DwC
- 06:11 PM Revision 936: Added ch_root_via to transform a map spreadsheet to use a different root, using a connecting root that links the input and output roots together
- 06:02 PM Revision 935: Added cols to select columns from a spreadsheet
- 06:02 PM Revision 934: util.py: Added list_subset()
- 05:42 PM Revision 933: ch_root: Fixed detection of unset env vars so that usage message is printed when any option is missing
- 05:41 PM Revision 932: opts.py: Call an error handler if an env var isn't set
- 05:40 PM Revision 931: util.py: Added function wrappers for statements noop() and and_()
- 05:07 PM Revision 930: inputs/NYBG-CSV: Map via DwC
- 04:50 PM Revision 929: Added subtract to subtract map spreadsheets
- 04:49 PM Revision 928: ch_root: Ignore empty lines
- 04:06 PM Revision 927: Added intersect to intersect two map spreadsheets
- 03:48 PM Revision 926: union: Clarified overwrite order of inputs in description
- 03:31 PM Revision 925: Removed no longer needed mappings/review
- 03:31 PM Revision 924: mappings/Makefile: Regenerate for_review maps automatically when a map changes
- 03:17 PM Revision 923: mappings/review: Generalized to convert all mappings to VegBIEN, not just a specific listed set (which was out of date)
- 03:16 PM Revision 922: mappings/for_review/DwC-VegBIEN.specimens.csv: Regenerated
- 03:12 PM Revision 921: inputs/NYBG/maps/DwC.specimens.csv: Fixed CollectedDate mapping to use the _date XML func
- 03:07 PM Revision 920: DwC mappings: Mapped Substrate and Vegetation
- 02:54 PM Revision 919: DwC mappings: Mapped BoundingBox, footprintWKT to location.dsgpoly
- 02:44 PM Revision 918: DwC mappings: Mapped Notes and PlantFungusDescription to bien.specimenDescription, merged together
- 02:41 PM Revision 917: xml_func.py: Added _merge and _label XML funcs
- 02:12 PM Revision 916: vegbien.sql: Added specimenreplicate.description
- 02:01 PM Revision 915: DwC mappings: Corrected mappings for CollectorNumber, individualID, recordNumber, and fieldNumber
- 01:31 PM Revision 914: DwC mappings: Mapped infraspecificEpithet to/from Subspecies
- 01:22 PM Revision 913: DwC-VegBIEN mapping: merge decimalLatitude/Longitude with verbatimLatitude/Longitude
02/10/2012
- 05:55 PM Revision 912: Regenerated vegbien.ERD exports
- 05:53 PM Revision 911: vegbien.ERD.mwb: Fixed lines. Repositioned elements.
- 05:41 PM Revision 910: vegbien.sql: Added locationevent.parent_id for subplot events to point to their parent plot event
- 05:21 PM Revision 909: vegbien.sql: Added plantconceptscope.namedplace_id
- 05:14 PM Revision 908: vegbien.sql: Added taxonoccurrence.method_id
- 05:00 PM Revision 907: vegbien.sql: Renamed plotmethod to method
- 04:47 PM Revision 906: vegbien.sql: Added locationdetermination.identifier_id
- 04:39 PM Revision 905: vegbien.ERD.mwb: Fixed lines
- 04:35 PM Revision 904: VegBIEN: Renamed locationplace to locationdetermination
- 04:18 PM Revision 903: vegbien.sql: Removed morphospecies, instead using plantconcept and new table plantconceptscope to represent scoped morphospecies. Reorganized plant* tables in ERD.
- 03:41 PM Revision 902: vegbien.ERD.mwb: Added plantcorrelation table to ERD
02/09/2012
- 07:08 PM Revision 901: mappings/VegX-VegBIEN.organisms.csv: Fixed TaxonomicRankBelowSubspeciesEnum mapping to use plantname instead of placing the name directly inside plantconcept. This bug was probably left over from the switch from linking directly to plantname rather than to a plantconcept which has a plantname.
- 06:54 PM Revision 900: Renamed get_errors to error_stats and added counts of each error message
- 06:24 PM Revision 899: Regenerated vegbien.ERD exports
- 06:22 PM Revision 898: vegbien.sql: Added plotmethod table
02/07/2012
- 05:22 PM Revision 897: input.Makefile: Renamed $(test) variable (containing the command to run a test) so it wouldn't conflict with test environment variable
- 05:10 PM Revision 896: mappings to VegBIEN: Don't map notes to taxondetermination because some organisms have notes but no species name
- 05:08 PM Revision 895: mappings/Makefile: Fixed bug in rules for cleaning up core maps whenever they change, which had the target and prerequisite reversed
- 04:39 PM Revision 894: input.Makefile: nolog option defaults to on when test is on
- 04:36 PM Revision 893: input.Makefile: Fixed bug where no log file was being created, even when nolog was off
- 04:11 PM Revision 892: Replaced all type(...) == str with util.is_str(...) to properly treat Unicode objects as strings
- 04:08 PM Revision 891: xml_dom.py: minidom.Element.write_opening(): Use new Attr.__str__() method for printing attributes
- 04:06 PM Revision 890: bin/map: For XML inputs, use xpath.get()'s new multiple result support to iterate over elements matching the root, rather than just iterating over the first matching element's siblings. This fixes the broken 2-step tests for stems, which didn't filter by the root's attributes.
- 04:04 PM Revision 889: xpath.py: get(): Fixed bug where limit was not passed to recursive get() calls
- 04:02 PM Revision 888: xml_dom.py: by_tag_name(): Iterate forwards over children unless last_only optimization turned on. Added Attr.__str__() and __repr__() for debug-printing Attrs.
- 03:10 PM Revision 887: xpath.py: get(): Turn off last_only optimization when limit > 1
- 03:07 PM Revision 886: xpath.py: get(): Added full support for returning multiple matches
- 02:33 PM Revision 885: xpath.py: get(): Added basic structure for returning multiple matches. Added limit parameter to select one or many matches.
- 02:13 PM Revision 884: input.Makefile: test/VegBIEN.%.2-step.xml: Use the core map specific to the test's table instead of the main core map
- 02:10 PM Revision 883: bin/map: Print error if map root not found inXML input
- 01:52 PM Revision 882: mappings/: Removed mappings used by old tester
- 01:48 PM Revision 881: inputs/SALVIAS/maps/VegX.*.csv: Replaced symlinks with actual files
- 01:45 PM Revision 880: Removed old tester
- 01:44 PM Revision 879: Switched to using inputs/test as main test target
- 01:38 PM Revision 878: Added SALVIAS-CSV tests
- 01:31 PM Revision 877: Added NYBG-CSV tests
- 01:28 PM Revision 876: input.Makefile: Run separate tests for each map spreadsheet (input table) rather than all tables at once. This will make it possible to test CSV inputs, which have one CSV per table.
- 12:54 PM Revision 875: Added NYBG-CSV input
- 12:47 PM Revision 874: inputs/Makefile: Fixed forwarding of empty targets to subdirs
- 12:20 PM Revision 873: Regenerated vegbien.ERD exports
- 12:17 PM Revision 872: vegbien.sql: Added morphospecies table
- 11:53 AM Revision 871: vegbien.ERD.mwb: Fixed lines
- 11:45 AM Revision 870: vegbien.sql: Removed the taxonOccurrence:aggregateOccurrence 1:1 constraint
02/06/2012
- 05:22 PM Revision 869: sql.py: truncate(): Use run_raw_query() instead of run_query() because truncate() does not use the recover functionality of run_query(). Also, in the profiling output, this separates the "normal" SQL statements (which use run_query()) from the "core" SQL statements (which use run_raw_query()).
- 05:09 PM Revision 868: vegbien.sql: Added indexes for each field in party used in duplicate elimination (for use by sql.put()'s DuplicateKeyException handler)
- 04:58 PM Revision 867: sql.py: run_raw_query(): In debug mode, print query after params have been substituted in
- 04:55 PM Revision 866: sql.py: Fixed index_cols() to handle UNIQUE indexes with expressions, whose column names are stored in a different format
- 04:04 PM Revision 865: sql.py: Print warning if SELECT statement missing a WHERE, LIMIT, or OFFSET clause. Changed bin/map DB input get-all-rows statement to pass start=0 to suppress this warning for that statement.
- 03:51 PM Revision 864: db_xml.py: Added start option to get() that passes through to sql.select()
- 03:50 PM Revision 863: sql.py: Added start option to select() to set the OFFSET
- 03:27 PM Revision 862: sql.py: If run_raw_query.debug flag is set, print each query executed (on a single line)
- 03:25 PM Revision 861: strings.py: Added one_line() function to make a string all on one line
- 03:20 PM Revision 860: strings.py: Renamed one_line() to remove_extra_newl() to better reflect what it does
- 03:19 PM Revision 859: bin/map: Don't print Done after an action in debug logging mode because it messes up newlines when more debugging info is printed right after it
- 02:45 PM Revision 858: input.Makefile: Added nolog option to disable creating a log file, e.g. for debugging runs
- 01:57 PM Revision 857: xml_dom.py: Remove extra newlines from single-line strings (bin/map doesn't need to do this itself anymore)
- 01:56 PM Revision 856: strings.py: Added is_multiline() and one_line() for removing extra newlines from single-line strings
- 01:48 PM Revision 855: bin/map: In debug mode, print input XPath's XML tree all on one line
- 01:22 PM Revision 854: sql.py: Switched try_insert() to use index_cols() instead of constraint_cols() for "duplicate key value violates unique constraint" errors because they can also be generated by UNIQUE indexes (and there is a UNIQUE index for every UNIQUE constraint)
- 01:14 PM Revision 853: sql.py: Added index_cols() to get cols used by an index (similar to constraint_cols())
- 01:00 PM Revision 852: vegbien.sql: Fixed duplicate elimination for party to use a UNIQUE index with COALESCE for nullable fields
- 01:00 PM Revision 851: sql.py: Fixed bug in try_insert() where DuplicateKeyException was passed only cols[0] instead of cols array
- 12:49 PM Revision 850: vegbien.ERD.mwb: Fixed lines
- 12:30 PM Revision 849: Added get_errors to select just the error messages from `map` output
- 11:59 AM Revision 848: Added profile_stats to analyze a profiling statistics file
- 11:59 AM Revision 847: bin/map: Added profile_to option which turns on profiling to the specified file
- 11:13 AM Revision 846: bin/map: Added "if __name__ == '__main__': main()" idiom so file can be included as well as run. This will be useful for profiling.
- 11:12 AM Revision 845: dates.py: Fixed strftime() to pad years and days with leading zeros as datetime.strftime() does
- 11:01 AM Revision 844: dates.py: Work around strftime() bug that can't deal with 2/29 on a leap year
- 10:44 AM Revision 843: xml_func.py: Added FormatException for SyntaxExceptions generated by strftime() (which are often Python bugs)
- 09:46 AM Revision 842: Added schemas/vegbank.revised.sql. Initial version has all "character varying" types replaced with text.
- 09:45 AM Revision 841: vegbien.sql: Replaced all "character varying" types with text, removing the length limits. Note that in PostgreSQL, text and "character varying" are stored the same way internally, so this does not affect performance or indexes.
02/04/2012
- 05:28 PM Revision 840: xml_dom.py: Added documentation labels to each section
- 05:21 PM Revision 839: xml_dom.py: Fixed bug in NodeTextEntryIter where an entry containing an element instead of a text node would be returned as the whole entry, instead of the value *of* the entry
- 04:56 PM Revision 838: bin/map: Added support for starting import at a specific row. Refactored row-processing code with and without a map to use a common process_rows() function (with the previous process_rows() being renamed to map_rows()).
- 04:14 PM Revision 837: bin/map: Use new util.cast()
- 04:14 PM Revision 836: util.py: Added cast() to cast a value while passing None through
- 04:09 PM Revision 835: bin/map: Print row # of rows with errors
- 03:22 PM Revision 834: sql.py: Fixed error in pkey() where recover was not passed as a named parameter to run_query()
- 03:12 PM Revision 833: sql.py: Added documentation labels to each section
- 03:10 PM Revision 832: sql.py: Added documentation labels to each section
- 03:03 PM Revision 831: db_xml.py: Used new sql.py recover functionality
- 03:03 PM Revision 830: sql.py: Added ability to recover from database errors so you don't get the error "InternalError: current transaction is aborted, commands ignored until end of transaction block"
- 02:25 PM Revision 829: vegbien.sql: Removed taxonoccurrence.taxoninferencearea because it's duplicated in aggregateoccurrence.inferencearea
- 02:18 PM Revision 828: bin/map: Highlight the "input row" and "output row:" labels in error messages
- 02:17 PM Revision 827: xml_func.py: Highlight nodes that were commented out because of errors
- 02:17 PM Revision 826: exc.py: Print exceptions with the first line highlighted in red
- 02:16 PM Revision 825: term.py: Added emph() and error()
- 01:51 PM Revision 824: vegbien.ERD.mwb: Fixed lines
- 01:47 PM Revision 823: vegbien.ERD.mwb: Fixed lines
- 01:46 PM Revision 822: vegbien.ERD.mwb: Added note, notelink, and revision
- 01:43 PM Revision 821: vegbien.ERD.mwb: Added embargo to diagram
- 01:27 PM Revision 820: Regenerated vegbien.ERD exports
- 01:24 PM Revision 819: vegbien.ERD.mwb: Fixed lines. Added "Core subset" and "Other tables" labels.
- 01:03 PM Revision 818: xml_func.py: Changed _date func to use new dates.strftime(), which can handle years before 1900
- 01:03 PM Revision 817: Added dates.py to handle date/time manipulation, such as fixing Python's broken strftime() that can't handle years before 1900
- 01:02 PM Revision 816: Regenerated mappings/for_review/DwC-VegBIEN.specimens.csv
02/03/2012
- 06:15 PM Revision 815: Regenerated vegbien.ERD exports
- 06:12 PM Revision 814: vegbien.ERD.mwb: Added reference and party tables
- 05:49 PM Revision 813: vegbien.ERD.mwb: Fixed lines
- 05:48 PM Revision 812: filter_ERD.csv: Remove fkeys to heavily-linked tables (reference, party)
- 05:30 PM Revision 811: Added to_do/milestones.doc
- 05:27 PM Revision 810: Renamed milestones.doc to timeline.doc
- 05:24 PM Revision 809: Added schemas/filter_ERD.csv and use it when generating vegbien.my.sql
- 05:16 PM Revision 808: vegbien.ERD.mwb: Added cover* to main diagram
- 04:53 PM Revision 807: vegbien.ERD.mwb: Started adding additional tables "below the fold" on the 2nd page
- 04:34 PM Revision 806: vegbien.ERD.mwb: Fixed lines
- 04:26 PM Revision 805: vegbien.ERD.mwb: Moved legend to top left to make room for more misc tables. Organized legend by location on diagram.
- 04:14 PM Revision 804: vegbien.ERD.mwb: Added soilobs table
- 04:07 PM Revision 803: vegbien.ERD.mwb: Added userdefined tables. Fixed lines.
- 03:57 PM Revision 802: vegbien.ERD.mwb: Changed location color to match VegBank ERD
- 03:53 PM Revision 801: vegbien.ERD.mwb: Added trait to diagram
- 03:38 PM Revision 800: vegbien.ERD.mwb: Added plantstatus to diagram. Added margins around diagram.
- 02:49 PM Revision 799: Added milestones.doc
- 12:39 PM Revision 798: DwC mappings: Fixed syntax of _date XML funcs to not wrap dates twice in a _date func
- 12:38 PM Revision 797: xml_func.py: Fixed bug in SyntaxException constructor where the cause was not passed to ExceptionWithCause
- 12:24 PM Revision 796: xml_dom.py: Override Node.__repr__ and Element.__repr__ to make sure self.toprettyxml() is used in all cases where a Node is converted to a string
- 11:57 AM Revision 795: util.py: Don't print type name in ConstraintError message because sometimes an entry tuple (e.g. from a dict) is checked, and then the type isn't useful
- 11:53 AM Revision 794: xml_func.py: Use conv_items(str, ...) in XML funcs that require strings, but don't otherwise use conv_items()
- 11:50 AM Revision 793: xml_func.py: Use xml_dom.TextEntryOnlyIter to check inputs for XML funcs that only handle strings, not whole elements
- 11:49 AM Revision 792: xml_dom.py: Added TextEntryOnlyIter to check that values returned by NodeTextEntryIter are strings from text nodes
- 11:48 AM Revision 791: util.py: Added is_str() and CheckedIter
01/31/2012
- 07:52 PM Revision 790: input.Makefile: Log each import to a new log file named according to the current time
- 07:38 PM Revision 789: xml_func.py: SyntaxException: Use exc.ExceptionWithCause
- 07:37 PM Revision 788: exc.py: Added str_() to convert an Exception to a string. Use str_() in print_ex().
- 07:31 PM Revision 787: xml_func.py: conv_items(): Only trap type conversion errors, not errors in map_items(). A previous bug incorrectly interpreted an error caught by map_items() as a SyntaxException.
- 07:23 PM Revision 786: xml_func.py: _date func: Fixed bug where setting defaults of a date part and removing zeros needed to happen after elements were converted to int. Refactored to use conv_items().
- 06:58 PM Revision 785: Makefiles: Added documentation labels to each section
- 06:55 PM Revision 784: Makefile: Added documentation labels to each section
- 06:49 PM Revision 783: input.Makefile: Added documentation labels to each section
- 06:44 PM Revision 782: xml_dom.py: Fixed conversion of minidom.Node to a string to use toprettyxml() again
- 06:40 PM Revision 781: Added inputs/SALVIAS-CSV/maps/VegBIEN.* maps
- 06:39 PM Revision 780: input.Makefile: If a test doesn't yet have accepted output, print its current output
- 06:24 PM Revision 779: mappings/Makefile: Only clean up a core map (intermediate format->VegBIEN) when it's modified. Note that the .last_cleanup files need to be under version control to prevent cleanups upon running any map command in a new checkout.
- 06:07 PM Revision 778: xml_func.py: Support dates with year/month/day == 0
- 06:06 PM Revision 777: input.Makefile: Print informative message if test failure is ignored
- 05:47 PM Revision 776: input.Makefile: Print path to test output so you can tell which directory a test is running in
- 05:45 PM Revision 775: input.Makefile: Fixed 2-step tests, which were still using $(root)/map when all bin/map calls were supposed to be routed through $(map)
- 05:05 PM Revision 774: xml_dom.py: Print simple XML elements on one line
- 04:33 PM Revision 773: Added inputs/SALVIAS-CSV/maps
- 04:33 PM Revision 772: input.Makefile: Added import support for CSV and XML inputs (no test support yet)
- 03:48 PM Revision 771: inputs/SALVIAS-CSV: Added src files
- 03:47 PM Revision 770: inputs/SALVIAS-CSV: Added src files
- 03:42 PM Revision 769: inputs: Removed no longer needed svn:ignore
- 03:41 PM Revision 768: input.Makefile: Coded svn:ignore directly in Makefile
- 03:40 PM Revision 767: inputs/Makefile: Added %-add target to add new inputs
- 03:35 PM Revision 766: input.Makefile: Added svn_props rule for updating svn:ignore of dir and subdirs
- 02:42 PM Revision 765: vegbien.sql: Made aggregateoccurrence:taxonoccurrence relationship 1:1
- 02:03 PM Revision 764: Added inputs/CTFS/src, with subset of files that will be in version control
- 02:01 PM Revision 763: mappings/for_review/DwC-VegBIEN.specimens.csv: Updated
01/30/2012
- 07:14 PM Revision 762: inputs/NYBG: Map via DwC instead of VegX
- 07:12 PM Revision 761: mappings/DwC-VegBIEN.specimens.csv: Provide schema URLs for namespaces (in the column header)
- 07:11 PM Revision 760: input.Makefile: Deal with core maps (the main *-VegBIEN map) whose roots are other than organisms
- 07:09 PM Revision 759: bin/map: In the xml_func.process() for XML outputs, pass an on_error that outputs to ex_tracker.track()
- 07:08 PM Revision 758: xpath.py: set_id(): Deal with XPaths with an
- 07:07 PM Revision 757: xml_dom.py: replace_with_text(): Accept Node inputs for cases where e.g. an XML func with an error has been replaced with a comment
- 07:06 PM Revision 756: xml_func.py: _date: Year defaults to 1900
- 05:18 PM Revision 755: DwC-VegBIEN.specimens.csv: Output DwC as XML instead of CSV
- 05:05 PM Revision 754: xpath.py: Allow XPath names to be quoted literals, just like values. Handle : embedded in a name (even without quotes), often indicating a namespace.
- 04:55 PM Revision 753: Removed no longer needed lib/ERD-wiki.csv
- 04:54 PM Revision 752: Parser.py: In syntax errors, only print <END> at end of string if error was *at* end of string
- 04:50 PM Revision 751: bin/map: Print error message rather than assertion failure for CSV output, which isn't supported yet
- 04:47 PM Revision 750: input.Makefile: Don't abort tester if via test fails, because CSV output is not supported yet
- 04:02 PM Revision 749: input.Makefile: Deal properly with more than one format that's being mapped via (e.g. DwC and VegX)
- 03:49 PM Revision 748: Renamed mappings/DwC-VegBIEN.organisms.csv back to mappings/DwC-VegBIEN.specimens.csv because it now uses /specimenreplicate as the output root
- 03:43 PM Revision 747: Renamed mappings/DwC-VegBIEN.organisms.csv back to mappings/DwC-VegBIEN.specimens.csv because it now uses /specimenreplicate as the output root
- 03:41 PM Revision 746: mappings/DwC-VegBIEN.organisms.csv: Fixed syntax error in a mapping
- 03:35 PM Revision 745: mappings/DwC-VegBIEN.organisms.csv: Sorted
- 03:34 PM Revision 744: mappings/Makefile: Sort DwC-VegBIEN.organisms.csv
- 03:33 PM Revision 743: inputs/NYBG/maps/VegBIEN.organisms.csv: Regenerated
- 03:12 PM Revision 742: mappings/DwC-VegBIEN.organisms.csv: Sort on output column
- 03:04 PM Revision 741: Renamed mappings/DwC-VegBIEN.specimens.csv to DwC-VegBIEN.organisms.csv because it (currently) uses /taxonoccurrence as the output root
- 03:03 PM Revision 740: Renamed mappings/DwC-VegBIEN.specimens.csv to DwC-VegBIEN.organisms.csv because it (currently) uses /taxonoccurrence as the output root
- 03:01 PM Revision 739: Makefile: Support recursing into subdir when no target is specified (e.g. make mappings/)
- 02:55 PM Revision 738: union: Merge rows (and column labels) using maps.merge_mappings()
- 02:51 PM Revision 737: join: Merge the column labels as well
- 02:51 PM Revision 736: maps.py: Eliminate duplicates when merging values in the same column
- 02:06 PM Revision 735: join: Moved mappings-specific merge functionality into maps.merge_mappings()
- 01:55 PM Revision 734: join: Use merge_rows() from new maps.py
- 01:54 PM Revision 733: Added new library maps.py for map spreadsheet manipulation
- 01:38 PM Revision 732: join: Merge comments of input map and join map
- 01:37 PM Revision 731: join: Report which input mappings are missing a mapping in the join map
- 01:03 PM Revision 730: join: Report which input mappings are missing a mapping in the join map
- 12:28 PM Revision 729: inputs/NYBG/maps/VegX.organisms.csv: Added note that the primary key has NULL values in some rows
- 12:24 PM Revision 728: input.Makefile: Preserve as many intermediate files as possible (make likes to delete intermediates if they aren't marked as .PRECIOUS)
- 11:46 AM Revision 727: sort: Sort empty strings last so that inputs with no mapping go at the end of the map spreadsheet
- 11:24 AM Revision 726: VegBIEN-VegBank.csv: Updated for recent table renames
- 11:21 AM Revision 725: join: For input mappings with no match in the join map, include them in the output map with an empty mapping
- 11:05 AM Revision 724: input.Makefile: Generalized to handle mapping via any format, not just VegX
- 10:44 AM Revision 723: input.Makefile: Don't print message to accept output for failing 2-step tests, because they use another test's accepted output
01/28/2012
- 06:02 PM Revision 722: input.Makefile: Don't abort tester if only 2-step test fails, as it's often finicky
- 05:55 PM Revision 721: xml_func.py: Raise xml_func.SyntaxException for ValueErrors generated by date.strftime() (e.g. year out of range due to poor Y2K support in some OS implementations of strftime)
- 05:51 PM Revision 720: xml_func.py: Raise xml_func.SyntaxException for ValueErrors generated by datetime.date() (e.g. month out of range)
- 05:46 PM Revision 719: vegbien.sql: Added project.reference_id to namespace project names by datasource
- 05:39 PM Revision 718: input.Makefile: Import all tables at once by default
- 05:11 PM Revision 717: bin/map: Print "Inserted ... new rows into database" message to stdout rather than stderr so it can be stored in the test case output as a validation check
- 05:08 PM Revision 716: Accepted initial test output for NYBG/test/import.out.ref
- 05:06 PM Revision 715: bin/map: Clean up datasource input values
- 05:06 PM Revision 714: strings.py: Added std_newl() to convert line endings and cleanup() to process strings with extra or nonstandard whitespace
- 04:46 PM Revision 713: PostgreSQL-MySQL.csv: Deal with custom types
- 04:36 PM Revision 712: vegbien.sql: Added aggregateoccurrence.occurrencestatus_dwc field
- 04:22 PM Revision 711: Regenerated vegbien.ERD exports
- 04:21 PM Revision 710: vegbien.ERD.mwb: Added commclass table to ERD
- 04:11 PM Revision 709: vegbien.sql: Removed direct pointer from location to namedplace because locationplace already has this relationship and we don't want to have an extra pointer just for duplicate elimination
- 03:56 PM Revision 708: vegbien.ERD.mwb: Added stratummethod to ERD
- 03:53 PM Revision 707: vegbien.sql: Removed locationevent.stratummethod_id because the stratummethod is a per-stratum (or technically, per-stratumtype) field
- 03:28 PM Revision 706: PostgreSQL-MySQL.csv: Remove CHECK constraints
- 03:20 PM Revision 705: PostgreSQL-MySQL.csv: Remove functions and triggers
- 03:13 PM Revision 704: vegbien.sql: Ensure that aggregateoccurrence.count == 1 when the aggregateoccurrence has a plantobservation. Use a trigger to do this automatically.
- 02:51 PM Revision 703: README.TXT: Added command for reimporting data
- 02:48 PM Revision 702: README.TXT: Added instructions to sync ERD with vegbien.sql schema. Organized commands into categories.
- 02:23 PM Revision 701: Regenerated vegbien.ERD exports
- 02:22 PM Revision 700: Added BIEN_logo.png
- 02:19 PM Revision 699: vegbien.ERD.mwb: Added color group legend
- 01:59 PM Revision 698: Regenerated vegbien.ERD exports
- 01:58 PM Revision 697: vegbien.ERD.mwb: Fixed lines
- 01:57 PM Revision 696: Regenerated vegbien.ERD exports
- 01:57 PM Revision 695: vegbien.ERD.mwb: Fixed lines and moved plant to its own color category
- 01:52 PM Revision 694: vegbien.ERD.mwb: Added colors to ERD
- 01:34 PM Revision 693: vegbien.ERD.mwb: Simplified diagram by removing column types
- 12:56 PM Revision 692: schemas/Makefile: Don't generate for_ERD DDLs because the ERD is now synced with the full schema
- 12:54 PM Revision 691: vegbien.ERD.mwb: Synced with whole schema
- 12:54 PM Revision 690: vegbien.sql: Reordered fields in tables truncated in the ERD so that all removed fields are at the end of the table
- 12:44 PM Revision 689: schemas/Makefile: Generate MySQL version of vegbien.sql as well as vegbien.for_ERD.sql for eventual use in syncing the ERD with the whole schema
- 12:43 PM Revision 688: PostgreSQL-MySQL.csv: Added translations for syntaxes used by pg_dump
- 12:41 PM Revision 687: PostgreSQL-MySQL.csv: Added translations for syntaxes used by pg_dump
- 12:40 PM Revision 686: repl: All regexps are by default in multiline and ignore case mode
01/27/2012
- 04:59 PM Revision 685: Regenerated vegbien.ERD exports
- 04:59 PM Revision 684: vegbien.sql: Made planttag a child of plantobservation instead of plant, since tags change over time
- 04:39 PM Revision 683: Regenerated vegbien.ERD exports
- 04:37 PM Revision 682: vegbien.sql: Removed no longer used plantobservation.aggregateoccurrence_id
- 04:35 PM Revision 681: VegX-VegBIEN mapping: Link aggregateoccurrence to plantobservation via forward pointer rather than backward child-to-parent pointer
- 04:32 PM Revision 680: vegbien.sql: Made plantobservation.aggregateoccurrence_id optional because link will soon be going in the other direction
- 04:25 PM Revision 679: vegbien.sql: Removed taxonbinmethod table since its fields are now in aggregateoccurrence
- 04:14 PM Revision 678: vegbien.sql: Added taxonbinmethod fields to aggregateoccurrence
- 04:10 PM Revision 677: vegbien.sql: Added back aggregateoccurrence.stratum_id
- 04:05 PM Revision 676: vegbien.sql: Added stratum.area
- 04:03 PM Revision 675: vegbien.sql: Removed denormalized duplicate fields from stratum
- 03:58 PM Revision 674: vegbien.sql: Added plant and planttag tables
- 03:43 PM Revision 673: VegBIEN: Renamed stem to stemobservation
- 03:35 PM Revision 672: vegbien.sql: Removed specimenreplicate:taxonoccurrence 1:1 requirement
- 03:32 PM Revision 671: VegBIEN: Renamed individualplant to plantobservation
- 03:22 PM Revision 670: vegbien.sql: Updated table comments for specimenreplicate and specimen
- 03:19 PM Revision 669: vegbien.sql: Added specimen table to tie specimenreplicates together
- 03:12 PM Revision 668: VegBIEN: Renamed specimen to specimenreplicate
- 03:06 PM Revision 667: Remerged ERD DDL into ERD
- 02:55 PM Revision 666: Redoing commit that linked aggregateoccurrence forward to individualplant, allowing many taxonoccurrences (e.g. one for each specimen) to point to the same plant (e.g. that those specimens came from)
- 12:57 PM Revision 665: Added NYBG input
- 12:48 PM Revision 664: input.Makefile: Run tests with verbose output
- 12:48 PM Revision 663: bin/map: Fixed bug where verbose/debug flags were ignored and message were always printed.
- 12:39 PM Revision 662: bin/map: Added verbose and debug options. Added initial debug info.
- 12:39 PM Revision 661: xml_dom.py: Added is_simple() to determine whether every child recursively has no more than one child. Used is_simple() to print condensed XML when simple nodes are converted to a string.
- 11:58 AM Revision 660: vegbien.sql: Enforce 1:1 relationship between aggregateoccurrence<->individualplant and taxonoccurrence<->specimen
- 11:44 AM Revision 659: vegbien.sql: Changed individualplant UNIQUE constraint to enforce 1:1 relationship between aggregateoccurrence and individualplant
- 11:35 AM Revision 658: Undoing previous commit since it would prevent a plant from being tied to a data source, because the aggregateoccurrence pointer goes in the wrong direction
- 11:18 AM Revision 657: vegbien.sql: Added aggregateoccurrence.individualplant_id to make a 1:1 relationship between aggregateoccurrence and individualplant
- 01:55 AM Revision 656: input.Makefile: Generate VegBIEN.2-step.xml correctly from VegX.xml, by removing DB config env vars passed to map for that test case. Note that this causes the VegBIEN.2-step.xml test to fail, because the 2-step mapping does not yet match the 1-step mapping.
- 01:28 AM Revision 655: input.Makefile: Don't need to filter test output since stderr now goes to the screen
- 01:21 AM Revision 654: input.Makefile: Don't save *.err outputs for each test because this information is printed to the screen
Also available in: Atom