Project

General

Profile

Statistics
| Revision:
  • svn:ignore: *

# Date Author Comment
12018 02/02/2014 12:49 AM Aaron Marcuse-Kubitza

inputs/input.Makefile: add!: verify/: also svn:ignore *.tsv, *.txt

11970 01/20/2014 11:33 AM Aaron Marcuse-Kubitza

moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).

11396 10/21/2013 07:14 PM Aaron Marcuse-Kubitza

fix: bin/map: put template: comment out the "Put template:" label so that the output is valid XML, and displays properly in a browser rather than showing a syntax error

11107 09/29/2013 08:58 PM Aaron Marcuse-Kubitza

bugfix: mappings/VegCore-VegBIEN.csv: nest all taxonoccurrences inside a stratum event, so that the parent locationevent is always fully populated before child locationevents point to it. (previously, a stub parent event was created when the child event was imported first, which blocked the fully-populated parent event from being inserted later on.) this uses auto-folding (for VegBank/CVS) and auto-forwarding (for other datasources) to prune empty stratum events for taxonoccurrences that don't have strata. (see wiki.vegpath.org/Auto-folding, wiki.vegpath.org/Auto-forwarding for more info about these normalization techniques.) note that the inserted row counts stay exactly the same for all datasources except VegBank (which was being fixed), indicating that this signficant change to the mappings did not change the semantics of the import of taxonoccurrences.

10866 09/04/2013 11:06 PM Aaron Marcuse-Kubitza

inputs/*/*/test.xml.ref: updated source.shortname for new datasource name, which now starts out with .new suffix

10565 08/01/2013 02:56 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: mapped municipality

10535 07/31/2013 04:56 PM Aaron Marcuse-Kubitza

inputs/HVAA/: switched to new-style import, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource

10534 07/31/2013 04:52 PM Aaron Marcuse-Kubitza

inputs/HVAA/Specimen/map.csv: fieldNumber: remapped to UNUSED

10533 07/31/2013 04:32 PM Aaron Marcuse-Kubitza

inputs/HVAA/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

10532 07/31/2013 04:31 PM Aaron Marcuse-Kubitza

inputs/HVAA/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

10531 07/31/2013 04:25 PM Aaron Marcuse-Kubitza

inputs/HVAA/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns

10425 07/25/2013 07:34 PM Aaron Marcuse-Kubitza

bugfix: inputs/*/*/map.csv for specimen tables: remapped eventDate,day,month,year to *Collected, because a general date always applies to the observation itself rather than to any parent event (specimens don't have a parent event)

10257 07/11/2013 12:09 PM Aaron Marcuse-Kubitza

inputs/*/*/map.csv: added distinguishing #... suffix (e.g. UNUSED#institutionID) to the special terms OMIT, PRIVATE, UNUSED (VegCore.vegpath.org#Special-terms) to avoid creating a collision in the staging table renaming

10242 07/10/2013 10:07 PM Aaron Marcuse-Kubitza

inputs/*/Source/VegBIEN.csv: regenerated for new-style import, which uses a symlink to mappings/VegCore-VegBIEN.csv instead of a custom mapping using the original column names

10209 07/10/2013 02:32 AM Aaron Marcuse-Kubitza

inputs/*/*/map.csv for CSV tables with a row_num column: added missing row_num entry, which is needed by the staging table column renaming to make the order of the map.csv columns match the order in the staging table

10199 07/09/2013 04:44 PM Aaron Marcuse-Kubitza

bugfix: inputs/*/Source/map.csv: added missing row_num entry, which is needed by the staging table column renaming to make the order of the map.csv columns match the order in the staging table. the staging table column renaming is now used by all Source tables.

10192 07/09/2013 01:04 AM Aaron Marcuse-Kubitza

bugfix: inputs/*/Source/: added missing ./run, which creates the new-style staging tables with the metadata fields as part of the table. this is needed now that these subdirs use installed staging tables instead of metadata-only map.csvs.

10179 07/06/2013 05:39 PM Aaron Marcuse-Kubitza

inputs/*/: added table.run for use by the table subdirs in new-style import. datasources without table subdirs do not need this.

10177 07/06/2013 05:17 PM Aaron Marcuse-Kubitza

inputs/*/: added top-level Makefile which includes inputs/input.Makefile, so that make can be run directly on the datasrc dir without needing to specify `--makefile=../input.Makefile` (see input.Makefile $(selfMake))

10170 07/06/2013 02:26 PM Aaron Marcuse-Kubitza

bugfix: inputs/*/Source/: use installed staging table (with blank-line data.csv) in order to also work with new-style import. this also fixes a benign diff between the by-row and by-col test outputs, where row-based import would not import the Source/ entries because there was not at least one row in the input. note that in order to ensure that all datasources are properly run, you need to check `svn st|sort` against the datasource schema names to see if any are missing.

10091 06/27/2013 12:28 PM Aaron Marcuse-Kubitza

added inputs/*/*/header.csv for CSV inputs, which are now generated by inputs/input.Makefile %/install

9882 06/12/2013 10:49 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: genus->taxonlabel.taxonomicname: filter out genera that contain numbers (using new _filter_genus()), which break TNRS and prevent it from matching any other parts of the name. later, these genera can instead be moved to the end of the name, where TNRS will correctly match them as Unmatched_terms.

8801 05/02/2013 08:53 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: SVN: add, %/add: */logs: also svn:ignore *.gz, used for compressed log files

8799 05/02/2013 08:50 PM Aaron Marcuse-Kubitza

added inputs/HVAA/Specimen/Herbario_occur_1360871068.csv.md5

8176 03/25/2013 09:01 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: %/.map.csv.last_cleanup: Run fix_line_endings after canon/translate to standardize Python's \r\n line endings back to \n. This prevents issues with mixed line endings because LibreOffice (and probably Excel) treat all cell-internal line endings as \n but row line endings as whatever the file had, while text editors like jEdit translate all line endings to whatever the autodetected line ending is. (This creates spurious line ending diffs when a map spreadsheet containing multiline cells is edited in a text editor.)

8078 03/19/2013 02:28 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: locationRemarks: Remapped to locationnarrative because location.notespublic is a boolean field

7841 03/05/2013 09:07 PM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Renamed specimenHolders to specimenHolderInstitutions to make it obvious that this is a list of institutions, such as would be in institutionCode in a DwC export.

7809 03/05/2013 03:02 AM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Editing: Changed ambiguous terms templates to list the name of the alternative itself in the alternatives list, instead of the ambiguous term + a source suffix. This avoids the need to choose a source suffix for each alternative of an ambiguous term, as the ambiguous term itself (without the suffix) can be listed as the synonym. Coordinates: Disambiguated verbatimCoordinateSystem.

7722 02/26/2013 07:11 PM Aaron Marcuse-Kubitza

inputs/HVAA/Specimen/map.csv: Remapped fieldNotes to UNUSED. This also fixes a bug where fieldNotes was colliding with other occurrenceRemarks inputs because it was missing a filter.

7716 02/26/2013 05:47 PM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Remapped organismNotes to be a synonym of occurrenceRemarks, since notes on an organism are more generally notes on an occurrence.

7715 02/26/2013 05:28 PM Aaron Marcuse-Kubitza

inputs/HVAA/Specimen/map.csv: Remapped dynamicProperties to occurrenceRemarks because it contains occurrenceRemarks data

7713 02/26/2013 05:19 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped occurrenceRemarks

7712 02/26/2013 04:53 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Only use one of collection and collectionURL instead of concatenating them when both are provided (as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#ACAD&gt;)

7703 02/26/2013 04:27 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: _avg(): Use numeric param names to work with SQL functions

7663 02/26/2013 10:08 AM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Source: Added authors. Renamed creator to firstPublisher. Moved dataOwner to Source as list. Coordinates: latitude/longitude, coordinatePrecision: Added units disambiguation. latitude/longitude_DMS: Added components. Global IDs: Changed "globally unique" tag to "global ID" and hyperlinked it to <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Global_IDs> (which now includes detailed examples of the various types of global IDs).

7661 02/26/2013 09:53 AM Aaron Marcuse-Kubitza

inputs/HVAA/Specimen/map.csv: Remapped coordinatePrecision to UNUSED

7646 02/22/2013 08:17 PM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Documentation has been added on how to choose term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming) and how to form globally unique ID values (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Forming-IDs). Source and Specimen terms have been renamed to be self-explanatory and unambiguous (the DwC equivalents remain as synonyms). Short definitions of Source terms have been added to explain the differences between them. Source, Specimen, and Collection terms have been shortened according to the new instructions for choosing preferred term names (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore#Naming).

7610 02/20/2013 07:33 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped reproductiveCondition

7570 02/16/2013 06:47 AM Aaron Marcuse-Kubitza

Added inputs/HVAA/Specimen/Herbario_occur_1360871068.csv.url

7555 02/14/2013 11:49 AM Aaron Marcuse-Kubitza

Added inputs/HVAA/