Project

General

Profile

Statistics
| Revision:

# Date Author Comment
4987 09/25/2012 07:22 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Removed unnecessary /_first/# suffix for multiple terms in the same _exists expression, because _exists() only checks whether its node is non-empty, and it does not matter how many child nodes it contains

4986 09/25/2012 06:57 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonoccurrence: taxonoccurrence_unique_within_locationevent unique index: Fixed bug where locationevent_id needed to be enclosed in COALESCE so that the unique constraint also applies to rows with NULL locationevent_ids (there is no other unique constraint handling these rows)

4985 09/25/2012 06:52 PM Aaron Marcuse-Kubitza

README.TXT: Documented that if the row-based and column-based imports produce different inserted row counts, this usually means that a table is underconstrained (the unique indexes don't cover all possible rows). The inserted row count difference occurs because column-based import collapses empty table rows into one insert, while row-based import performs an insert of the empty row for each input row. Without a unique index to combine multiple row-based inserts, extra rows will be added.

4984 09/25/2012 06:48 PM Aaron Marcuse-Kubitza

sql_io.py: put_table(): Warn if inserting empty table rows

4983 09/25/2012 06:13 PM Aaron Marcuse-Kubitza

schemas/py_functions.sql: _namePart(): Fixed bug where it was returning the empty string instead of NULL

4982 09/25/2012 05:57 PM Aaron Marcuse-Kubitza

schemas/functions.sql, py_functions.sql: Added schema comment that functions must always return NULL in place of the empty string, to ensure that empty strings do not find their way into VegBIEN. Note that row-based import automatically removes empty strings because the intermediate values are stored in XML and our XML DOM traversing code auto-replaces the empty string with NULL. Column-based import, on the other hand, does not, because the intermediate data is stored in database temp tables instead of a DOM tree.

4981 09/25/2012 05:31 PM Aaron Marcuse-Kubitza

root map: Fixed custom public schema override to work with schemas lists that include public, by replacing public with the new public schema instead of just appending it

4980 09/25/2012 04:53 PM Aaron Marcuse-Kubitza

inputs/*/*/map.csv: Prefix a * to every term that's not in Veg+ for easy identification of unmapped terms when editing map.csv. Note that canon will remove the * when it finds a matching Veg+ term.

4979 09/25/2012 04:52 PM Aaron Marcuse-Kubitza

inputs/*/*/map.csv: Prefix a * to every term that's not in Veg+ for easy identification of unmapped terms when editing map.csv. Note that canon will remove the * when it finds a matching Veg+ term.

4978 09/25/2012 04:36 PM Aaron Marcuse-Kubitza

ins_col: Added column fill value param

4977 09/25/2012 04:16 PM Aaron Marcuse-Kubitza

inputs/VegBank/stemcount/map.csv: Fixed bug where taxonimportance_id needed to point to aggregateOccurrenceID instead of taxonOccurrenceID

4976 09/25/2012 04:15 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Don't forward individualID to taxonoccurrence.sourceaccessioncode when aggregateOccurrenceID is present

4975 09/25/2012 03:52 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

4974 09/24/2012 06:45 PM Aaron Marcuse-Kubitza

Regenerated vegbien.ERD exports

4973 09/24/2012 06:33 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: placepath.otherranks comment: Added analogous text from taxonpath.otherranks

4972 09/24/2012 06:31 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonpath.author comment: Added equivalent Darwin Core term

4971 09/24/2012 06:27 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxon columns: Added descriptive comments for data dictionary

4970 09/24/2012 06:15 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: placepath: Added canon_placepath_id, analogous to taxonpath.canon_taxonpath_id

4969 09/24/2012 06:09 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: place, placepath descriptive comments: Added analogous text from taxon/taxonpath

4968 09/24/2012 06:05 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonpath: descriptive comment: Changed "applicable taxon" to "identified taxon"

4967 09/24/2012 05:58 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxon: descriptive comment: Reworded to emphasize that this stores only one rank (e.g. family) of the full taxonomic name, in contrast to taxonpath, which stores all of them

4966 09/24/2012 05:54 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonpath: descriptive comment: Clarified that this is the full path to a taxon, including all components of the taxonomic name

4965 09/24/2012 05:48 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Replaced "scientific name" with "taxonomic name" for schema-wide consistency and for consistency with the taxon/taxonomic name vocabulary

4964 09/24/2012 05:38 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonpath named ranks: Added descriptive comments for data dictionary

4963 09/24/2012 05:34 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonpath columns other than named ranks: Added descriptive comments for data dictionary

4962 09/24/2012 05:14 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonscope: descriptive comment: Reworded to make the first sentence a noun, for consistency with other descriptive table comments

4961 09/24/2012 05:13 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxon: descriptive comment: Added note that the taxonname stores only one rank (e.g. family) of the full identifying name

4960 09/24/2012 05:07 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonpath: descriptive comment: Reworded to make the first sentence a noun, for consistency with other descriptive table comments. The convention is for the first "sentence" to be a noun which describes the entity that the table models.

4959 09/24/2012 05:00 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: comments: Removed units from comments on fields which already have a units suffix, to avoid having to keep the units in sync between the suffix and the comment. Note that the units were abbreviated equally in the suffixes and comments, so this did not result in a loss of information other than the ^ for a quantity squared (but it's obvious enough that m2 is m^2).

4958 09/24/2012 04:54 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonscope: descriptive comment: Added period for consistency with other descriptive table comments

4957 09/24/2012 04:50 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxon: Added descriptive comment for data dictionary

4956 09/24/2012 04:48 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: VegBank-equivalent tables comments: Prepended "Equivalent to" before VegBank, so the equivalent tables statement can fit grammatically after a description of the table instead of having to be the first phrase in the descriptive table comment

4955 09/24/2012 04:41 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxon: VegBank-equivalent tables comment: Added plantName and applicable columns from plantStatus, which are also part of the taxon table

4954 09/24/2012 04:37 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: placepath: Added otherranks field, analogous to taxonpath.otherranks

4953 09/24/2012 04:26 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonpath: Added descriptive comment for data dictionary

4952 09/24/2012 03:36 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

4951 09/24/2012 02:58 PM Aaron Marcuse-Kubitza

inputs/UNCC/Specimen/map.csv: accession: Documented that it's globally unique, although occasionally duplicated

4950 09/24/2012 02:54 PM Aaron Marcuse-Kubitza

inputs/REMIB/Specimen/map.csv: Remapped accession_number to catalogNumber because it is not globally unique, only (usually) unique within the institution providing the data ("acronym"). Note that there are nevertheless 11,869 rows where an accession_number appears multiple times within the same institution.

4949 09/24/2012 02:45 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Only use institutionCode+collectionCode+catalogNumber as the authorlocationcode (location-scoping ID) if there is actually a catalogNumber. Otherwise, the mapping process would attempt to create one location for each collection in the datasource, when there should be one location for each specimen.

4948 09/24/2012 02:36 PM Aaron Marcuse-Kubitza

schemas/py_functions.sql: _namePart(): Slice the first name from the beginning of the string to one word before the end, instead of one after the beginning, in order to avoid overlap with the last name, which starts one before the end, when there is only one word. Note that only one word means the name is assumed to be a last name. This assumption may not always be true, but when a datasource provides the name concatenated, an assumption must be made when not all name components are present.

4947 09/24/2012 02:30 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: party: Added check constraint to require at least an organizationname or surname. Previously, NULL entries for the collector or identifier incorrectly caused the creation of an empty party entry, hence the lower inserted row counts now that this is no longer created.

4946 09/24/2012 02:17 PM Aaron Marcuse-Kubitza

inputs/REMIB/Specimen/map.csv: Remapped acronym to institutionCode because this is an aggregator, and the field lists the datasource each record was aggregated from. Note that the inserted row count changes because of different duplicate elimination strategies in specimenreplicate and party (which institutionCode is placed in).

4945 09/24/2012 02:11 PM Aaron Marcuse-Kubitza

inputs/REMIB/Specimen/create.sql: Also filter out rows where acronym (collectionCode) is NULL because this is a required field for valid records

4944 09/24/2012 01:28 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonpath: Renamed scientificnameauthor to author so the column name doesn't have "scientificname" in it, which made the term look confusingly like scientificname itself. Added descriptive comment that this is the author of the scientific name.

4943 09/24/2012 01:19 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonpath: Renamed canon_id to canon_taxonpath_id to clarify that this is a recursive fkey. The convention is that a recursive fkey includes the table name plus a descriptive prefix.

4942 09/24/2012 01:14 PM Aaron Marcuse-Kubitza

schemas/filter_ERD.csv: Don't filter out fkeys from taxonpath to itself

4941 09/24/2012 11:32 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonpath: Added canon_id for the canonical (scrubbed) taxonpath determined by TNRS

4940 09/24/2012 11:24 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonpath: taxonpath_unique_within_datasource_by_name unique index: Added otherranks, so that ranks without a named column will be used in uniquely identifying the taxonpath

4939 09/24/2012 11:22 AM Aaron Marcuse-Kubitza

sql.py: DbConn.col_info(): Parse array types as sql_gen.ArrayType

4938 09/24/2012 11:22 AM Aaron Marcuse-Kubitza

sql_gen.py: EnsureNotNull: Support ArrayType types

4937 09/24/2012 11:21 AM Aaron Marcuse-Kubitza

strings.py: remove_prefix(), remove_suffix(): Added require param to raise aan exception if the string does not have the given prefix/suffix

4936 09/24/2012 11:06 AM Aaron Marcuse-Kubitza

sql.py: DbConn.col_info(): Moved parsing of user-defined datatypes to Python code, so that parsing for other composite types which also requires both data_type and udt_name can easily be added

4935 09/24/2012 11:03 AM Aaron Marcuse-Kubitza

sql_gen.py: Added ArrayType

4934 09/24/2012 10:29 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Scope taxonpath instead of taxon with taxonscope, because a morphospecies name is specific to a datasource entity, so it should go in the datasource-specific taxonpath table instead of the datasource-general taxon table

4933 09/24/2012 10:14 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: taxonpath: Added otherranks array column to store ranked names without a named column. Documented that ranks with no named column should be stored in this new field instead of in a chain of taxons pointed to by taxon_id. This ensures that only the tree of life uses the taxon table.

4932 09/24/2012 09:47 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Removed no longer used table stemtag, which has been replaced by stemobservation.tag, stemobservation.tags

4931 09/21/2012 04:28 PM Aaron Marcuse-Kubitza

inputs/ARIZ/Specimen/test.xml.ref: Updated after reinstalling staging table with new sql_io.null_strs

4930 09/21/2012 04:22 PM Aaron Marcuse-Kubitza

inputs/VegBank/: Added stemlocation/

4929 09/21/2012 04:17 PM Aaron Marcuse-Kubitza

inputs/VegBank/: Added stemcount/

4928 09/21/2012 04:10 PM Aaron Marcuse-Kubitza

sql_io.py: cleanup_table(): Fixed bug where couldn't run any update statement when no columns are text

4927 09/21/2012 03:57 PM Aaron Marcuse-Kubitza

csv2db: COPY FROM mode: Removed no longer needed explicit column list, now that the initial table has the exact width of the CSV (the row_num is added later)

4926 09/21/2012 03:55 PM Aaron Marcuse-Kubitza

csv2db: Add any row_num column after creating the table, so it does not interfere with row widths when using COPY FROM without explicit column names

4925 09/21/2012 03:48 PM Aaron Marcuse-Kubitza

csv2db: Fixed bug where tables without a row_num (such as *.src tables) were not properly supported when the CSV contained ragged rows, because the columns were truncated to # column names + 1 but there was no row_num to be the +1. This was solved by moving row_num to the end, so that it does not impact the column count whether it's there or not.

4924 09/21/2012 03:44 PM Aaron Marcuse-Kubitza

csv2db: Fixed bug where tables without a row_num (such as *.src tables) were not properly supported when the CSV contained ragged rows, because the columns were truncated to # column names + 1 but there was no row_num to be the +1. This was solved by moving row_num to the end, so that it does not impact the column count whether it's there or not.

4923 09/21/2012 03:28 PM Aaron Marcuse-Kubitza

inputs/VegBank/: Added taxonimportance/

4922 09/21/2012 03:20 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Added and mapped aggregateOccurrenceID

4921 09/21/2012 03:12 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: taxonOccurrenceID: Re-sourced to VegBank taxonobservation and DwC occurrenceID, because this is where the VegBIEN table name came from

4920 09/21/2012 02:57 PM Aaron Marcuse-Kubitza

tnrs_client: Support parsing multiple taxons at once, by specifying each as a command-line argument. Increased the max_pause to 10 min to support large batches. Limited the batch size to 5000 names, using the limit at <http://tnrs.iplantcollaborative.org/TNRSapp.html&gt;. Note that when using xargs to pass many names, xargs will by default split its arguments into chunks of 5000. You can change this using the -n option.

4919 09/21/2012 02:29 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

4918 09/21/2012 01:20 PM Aaron Marcuse-Kubitza

Added tnrs_client. Note that obtaining an actual CSV requires four (!) steps: submit, retrieve, prepare download, and download. The output of the retrieve step is unusable because the array has different lengths depending on the taxonomic ranks present in the provided taxon name. This initial version runs one name at a time, but could later be expanded to batch process because TNRS can run multiple names at once.

4917 09/21/2012 12:36 PM Aaron Marcuse-Kubitza

streams.py: Line iteration: Added read_all()

4916 09/21/2012 08:24 AM Aaron Marcuse-Kubitza

inputs/Madidi/Plot/map.csv: Soil component measurements: Documented that units are assumed to be % based on the range of values

4915 09/21/2012 08:18 AM Aaron Marcuse-Kubitza

sql_io.py: null_strs: Added '-'

4914 09/21/2012 08:18 AM Aaron Marcuse-Kubitza

sql_io.py: cleanup_table(): Fixed bug where each column name needed to be converted to Unicode before being concatenated with other strings, to support non-ASCII characters

4913 09/21/2012 07:57 AM Aaron Marcuse-Kubitza

inputs/SALVIAS/plotMetadata/map.csv, inputs/SALVIAS-CSV/Plot/map.csv: Soil component measurements: Documented that units are assumed to be % based on the range of values

4912 09/21/2012 07:52 AM Aaron Marcuse-Kubitza

inputs/SALVIAS/plotMetadata/map.csv, inputs/SALVIAS-CSV/Plot/map.csv: Soil component measurements: Removed no longer needed old-style _units filter, now that unit conversion is handled by mappings/VegCore-VegBIEN.csv using _percent_to_fraction

4911 09/21/2012 07:48 AM Aaron Marcuse-Kubitza

inputs/VegBank/observation_/map.csv: soilObs fields: Cited data dictionary source of units

4910 09/21/2012 07:15 AM Aaron Marcuse-Kubitza

mappings/Veg+-VegCore.csv: Soil component measurements: Added unitless terms that automap to all alternatives of units

4909 09/21/2012 07:08 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Added term with *_fraction units for every *_percent term

4908 09/21/2012 07:03 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Soil component measurements: Added default units of percent (cmol_kg for cationExchangeCapacity). This involves translating the names everywhere and adding a _percent_to_fraction conversion in mappings/VegCore-VegBIEN.csv.

4907 09/20/2012 11:15 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Remapped verbatimLatitude/Longitude to locationcoords.verbatimlatitude/longitude because these fields now contain only non-decimal coordinates. This involves removing the _alt suffix on decimalLatitude/Longitude, which causes the VegBIEN.csvs to change.

4906 09/20/2012 11:11 PM Aaron Marcuse-Kubitza

inputs/*/*/map.csv: Remapped latitude/longitude to decimalLatitude/Longitude because these fields almost always have units of decimal degrees

4905 09/20/2012 11:06 PM Aaron Marcuse-Kubitza

inputs/*/*/map.csv: Remapped latitude/longitude to decimalLatitude/Longitude because these fields almost always have units of decimal degrees

4904 09/20/2012 10:54 PM Aaron Marcuse-Kubitza

inputs/SpeciesLink/Specimen/map.csv: Documented that dwc_geospatial_VerbatimLatitude/Longitude contain a mix of DMS and other verbatim coordinates

4903 09/20/2012 10:47 PM Aaron Marcuse-Kubitza

inputs/QMOR/Specimen/map.csv: Remapped verbatimLatitude/verbatimLongitude to latitude_DMS/longitude_DMS since these fields contain DMS values

4902 09/20/2012 10:43 PM Aaron Marcuse-Kubitza

inputs/Madidi/Plot/map.csv: Remapped Latitude/Longitude (DMS) to new latitude_DMS/longitude_DMS

4901 09/20/2012 10:41 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped latitude_DMS, longitude_DMS

4900 09/20/2012 10:38 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Added latitude_DMS, longitude_DMS

4899 09/20/2012 10:34 PM Aaron Marcuse-Kubitza

inputs/REMIB/Specimen/map.csv: Remapped lat_deg/long_deg to decimalLatitude/Longitude because these values are (integer) degrees suitable for decimalLatitude/Longitude. Note that the other DMS fields are not yet translated to decimal degrees.

4898 09/20/2012 10:28 PM Aaron Marcuse-Kubitza

mappings/Veg+-VegCore.csv: Remapped latitude/longitude to decimalLatitude/Longitude because these fields almost always have units of decimal degrees

4897 09/20/2012 10:26 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Added empty mappings for special values (OMIT, etc.), so that they don't show up in **/unmapped_terms.csv. Note that the VegBIEN.csvs only change because the "No join mapping" errors change to "No non-empty join mapping".

4896 09/20/2012 10:23 PM Aaron Marcuse-Kubitza

input.Makefile: Maps validation: %/unmapped_terms.csv, %/new_terms.csv: Don't automatically regenerate the aggregated unmapped_terms.csv, new_terms.csv because this almost doubles the remake time when a mappings/ prerequisite changes (41s -> 75s)

4895 09/20/2012 10:14 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Added empty mappings for special values (OMIT, etc.), so that they don't show up in **/unmapped_terms.csv. Note that the VegBIEN.csvs only change because the "No join mapping" errors change to "No non-empty join mapping".

4894 09/20/2012 10:09 PM Aaron Marcuse-Kubitza

inputs/GBIF/Specimen/map.csv: Remapped VerbatimLatitude/Longitude to decimalLatitude/Longitude because DecimalLatitude/Longitude just contains VerbatimLatitude/Longitude cast to a low-resolution float, which created spurious repeating decimals

4893 09/20/2012 09:56 PM Aaron Marcuse-Kubitza

mappings/Makefile: .VegCore-VegBIEN.csv.last_cleanup: Generate VegCore-VegBIEN.unsourced_terms.csv whenever VegCore-VegBIEN.csv changes, to track VegCore terms that are mapped to VegBIEN but not documented in VegCore.csv. Note that this file is not svn:ignored, so it will show up with a ? when the user runs `svn st` if there are any unsourced terms.

4892 09/20/2012 09:47 PM Aaron Marcuse-Kubitza

mappings/Makefile: Changed catch-all `.%.last_cleanup: %` target to a specific target for VegCore-VegBIEN.csv, because it's the only file that uses this target

4891 09/20/2012 09:45 PM Aaron Marcuse-Kubitza

mappings/: Don't generate a for_review version of Veg+-VegCore.csv, because it is identical to the machine-readable Veg+-VegCore.csv (there are no output XPaths to simplify)

4890 09/20/2012 09:41 PM Aaron Marcuse-Kubitza

mappings/: Don't generate a for_review version of VegX-VegCore.csv, because it is identical to the machine-readable VegX-VegCore.csv (there are no output XPaths to simplify)

4889 09/20/2012 09:37 PM Aaron Marcuse-Kubitza

mappings/: Removed Veg+.unmapped_terms.csv because these terms are found in each datasource's new_terms.csv, which are updated regularly, while this file isn't, and which exist for every datasource, while this file only contained terms from a few datasources

4888 09/20/2012 09:29 PM Aaron Marcuse-Kubitza

inputs/ARIZ/Specimen/map.csv: Remapped VerbatimLatitude, VerbatimLongitude to UNUSED