Project

General

Profile

Statistics
| Revision:

# Date Author Comment
7504 02/07/2013 01:57 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: datasourceRecordID: Fixed bug where also need to add datasourceRecordID next to occurrenceID for an institutionCode remap switch

7503 02/07/2013 01:57 PM Aaron Marcuse-Kubitza

inputs/bien_web/observation/test.xml.ref: Regenerated

7502 02/07/2013 01:48 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times using the import_times bugfix for times longer than a day

7501 02/07/2013 01:45 PM Aaron Marcuse-Kubitza

import_times: times(): Fixed bug where need to match whitespace in times, in order to match times with days

7500 02/07/2013 12:00 PM Aaron Marcuse-Kubitza

inputs/*/Specimen/map.csv: Remapped ID to datasourceRecordID

7499 02/07/2013 11:55 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped datasourceRecordID

7498 02/07/2013 11:51 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

7497 02/07/2013 08:38 AM Aaron Marcuse-Kubitza

inputs/FIA/_src/_README.TXT: Documented that the refresh is missing some PLT_CN values present in the original version

7496 02/07/2013 08:33 AM Aaron Marcuse-Kubitza

inputs/FIA/import_order.txt: Reverted back to using FIA_COND_unique instead of COND_unique because the PLT_CN IDs in the refresh don't match the PLT_CN IDs in the original version, making COND_unique and Organism incompatible

7495 02/07/2013 08:27 AM Aaron Marcuse-Kubitza

inputs/FIA/import_order.txt: Removed FIA_COND_unique, which is superseded by COND_unique

7494 02/07/2013 08:26 AM Aaron Marcuse-Kubitza

inputs/FIA/import_order.txt: Fixed bug where need to import COND_unique before Organism because the plot entries need to be created before they can be linked to by organisms

7493 02/07/2013 07:25 AM Aaron Marcuse-Kubitza

redmine_synonyms: sed pattern: Match <h# directly at the beginning of the line rather than after ^.*, which greatly speeds up the pattern matching because the first character is a literal character. (If <h# were not located at the left margin, the ^.* would unfortunately still be needed because the beginning of the line needs to be matched in order to be removed by the replacement operation.)

7492 02/07/2013 07:22 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. Alternatives are now able to use h3 instead of h4 (which had display problems). realLatitude/Longitude is now no longer needs the ? prefix to have its replacement (PRIVATE) interpreted as an alternative, and thus is properly able to be included in the vocabulary.

7491 02/07/2013 07:16 AM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.vocab.csv: Use the term's type label instead of its header level to determine if it's a synonym or alternative. This allows header levels to be chosen for presentational reasons rather than being constrained by being parsable.

7490 02/07/2013 07:05 AM Aaron Marcuse-Kubitza

redmine_synonyms: Don't require ambiguous terms to start with ?, because the ambiguous term for an alternative can be identified simply by choosing the last term that didn't have a type label (previously, this would have been the last term that wasn't h3 or h4)

7489 02/07/2013 07:01 AM Aaron Marcuse-Kubitza

redmine_synonyms: Use the term's type label instead of its header level to determine if it's a synonym or alternative. This allows header levels to be chosen for presentational reasons rather than being constrained by being parsable.

7488 02/07/2013 06:26 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. The data dictionary has been reformatted to be much more vertically compact, by placing the term type (Synonym, Alternative, etc.) and sources (From:) on the same line as the term. Note that globalUniqueIdentifier_SpeciesLink has been removed from the vocabulary because a definition entry has been added for it (when this entry is missing, the term is incorrectly identified as a primary term).

7487 02/07/2013 06:21 AM Aaron Marcuse-Kubitza

mappings/Makefile, redmine_synonyms: Updated for new VegCore data dictionary format, which prefixes the term type (Synonym, Alternative, etc.) to the term instead of including it as a section label. This ensures that the term type of a non-primary term is shown next to the term when it is visited via a permalink, which causes the term header to appear at the top of the screen and obscures the section header containing the type.

7486 02/07/2013 06:00 AM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.thesaurus.csv: removal of tables: ignore errors if grep found no match

7485 02/07/2013 02:06 AM Aaron Marcuse-Kubitza

Renamed mappings/VegCore.csv to VegCore.vocab.csv and Veg+-VegCore.csv to VegCore.thesaurus.csv for clarity

7484 02/07/2013 02:03 AM Aaron Marcuse-Kubitza

mappings/Makefile, input.Makefile: Renamed $(dict) to $(thesaurus) because Veg+-VegCore.csv is actually a thesaurus, not a dictionary

7483 02/07/2013 01:57 AM Aaron Marcuse-Kubitza

mappings/Makefile: Replaced occurrences of VegCore.csv with $(vocab) and Veg+-VegCore.csv with $(dict)

7482 02/06/2013 07:34 PM Aaron Marcuse-Kubitza

README.TXT: Maintenance: VegCore data dictionary: When moving terms, check that no terms were lost: Updated steps now that VegCore.csv and Veg+-VegCore.csv are sorted by name, so that a comparison of added/deleted counts is not necessary and a simple `svn di` can be used

7481 02/06/2013 07:33 PM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Sort terms by name so that reordering terms in the VegCore data dictionary does not cause Veg+-VegCore.csv to change. This makes it much easier to identify synonyms and ambiguous terms that were accidentally deleted during a data dictionary refactoring. (Note that these are no longer included in VegCore.csv, so this is required in addition to sorting VegCore.csv by name.)

7480 02/06/2013 07:26 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Sort terms by name so that reordering terms in the VegCore data dictionary does not cause VegCore.csv to change. This makes it much easier to identify terms that were accidentally deleted during a data dictionary refactoring.

7479 02/05/2013 06:19 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. This adds cf_aff.

7478 02/05/2013 06:18 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Filter out namespaces by matching only terms whose header links within the data dictionary

7477 02/05/2013 06:08 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. This causes TNRS's Annotations (cf/aff) to be mapped into VegBIEN.

7476 02/05/2013 06:05 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: matched*Fit_fraction: Remapped to taxonconfidence instead of taxonfit

7475 02/05/2013 05:56 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Fixed bug where need to remove duplicates, which are no longer supported by canon, by removing alternatives of ambiguous terms when these occur separately from their definitions

7474 02/05/2013 05:29 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Removed synonyms and ambiguous terms, since the canonicalization of them is handled by Veg+-VegCore.csv. This also reduces the time it takes canon to build the in-memory Python dict of replacements, which scales to all inputs and should speed up the build/test command.

7473 02/05/2013 05:22 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Removed synonyms, since the canonicalization of them is handled by Veg+-VegCore.csv

7472 02/05/2013 05:10 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Match terms by header # instead of matching all anchors, in order to include the leading ? before an ambiguous term

7471 02/05/2013 04:42 PM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Generate dynamically from VegCore.htm, which allows the VegCore thesaurus to be automatically kept up to date. More importantly, it allows terms in all map spreadsheets to be updated simultaneously when a term is renamed (e.g. by replacing a term with one of its synonyms).

7470 02/05/2013 04:40 PM Aaron Marcuse-Kubitza

mappings/VegX-VegCore.csv: Applied term renamings from the new dynamically generated Veg+-VegCore.csv. Updates to VegCore term names that have occurred since the data dictionary was created are now able to take effect, which involves remapping several fields.

7469 02/05/2013 04:32 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv, inputs/*/*/map.csv: Applied term renamings from the new dynamically generated Veg+-VegCore.csv, which reflects the current state of the data dictionary. (Permanently switching to the new Veg+-VegCore.csv will be a separate change.) Updates to VegCore term names that have occurred since the data dictionary was created are now able to take effect, which involves remapping and inferring units on several fields.

7468 02/05/2013 04:27 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped basalDiameter_in

7467 02/05/2013 04:15 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped diameterBreastHeightGentry_cm, basalDiameter_cm, precipitation_mm

7466 02/05/2013 04:14 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Added _mm_to_m()

7465 02/05/2013 03:56 PM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Fixed bugs where also need to filter out ambiguous tables, but shouldn't filter out acronyms (which are regular fields)

7464 02/05/2013 03:40 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: locationID->location.sourceaccessioncode: Removed restriction that this mapping can't occur if geovalidation information is present. The locationID is no longer mapped to the place.sourceaccessioncode, so this filter is not necessary.

7463 02/05/2013 03:38 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7462 02/05/2013 03:19 PM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Fixed bug where need to filter out table names to avoid applying table replacements to fields which have the same name as a table

7461 02/05/2013 03:03 PM Aaron Marcuse-Kubitza

inputs/Madidi/map.csv: Fixed bug where needed to remove duplicate input names, now that translate doesn't allow them

7460 02/05/2013 01:59 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegX-VegCore.csv: Sort by the input column instead of the output column to keep the sort order stable across VegCore term renames

7459 02/05/2013 01:46 PM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Before running collapse_multimap, canonicalize alternatives of ambiguous terms using unambiguous mappings. This ensures that the alternatives lists contain only canonical VegCore terms rather than synonyms.

7458 02/05/2013 01:43 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. All synonyms are now hyperlinked, allowing them to be matched by redmine_synonyms.

7457 02/05/2013 01:31 PM Aaron Marcuse-Kubitza

mappings/Veg+-VegCore.csv: Removed Sources, Definition columns because source information is now in the VegCore data dictionary

7456 02/05/2013 01:25 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. Ambiguous terms newly available to redmine_synonyms due to the bugfix now have multiple alternatives.

7455 02/05/2013 01:25 PM Aaron Marcuse-Kubitza

redmine_synonyms: Ambiguous terms: Fixed bug where need to use header # instead of term name to determine whether a term is an alternative, because some alternatives (e.g. verbatimElevation) don't follow the units-suffix naming convention.

7454 02/05/2013 12:58 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. All ambiguous terms now have multiple alternatives, preventing them from being automapped to a single alternative without prompting the user for confirmation

7453 02/05/2013 12:50 PM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: translate: Fixed bug where need to run on $@ instead of $<

7452 02/05/2013 12:49 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. All ambiguous terms now have multiple alternatives, preventing them from being automapped to a single alternative without prompting the user for confirmation

7451 02/05/2013 12:22 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. All mappings/Veg+-VegCore.csv terms are now added as synonyms or separate terms.

7450 02/05/2013 10:26 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. Most ambiguous terms are now split into alternatives, and most mappings/Veg+-VegCore.csv terms are now added as synonyms.

7449 02/05/2013 06:12 AM Aaron Marcuse-Kubitza

canon: Raise an error if two input terms map to the same simplified string

7448 02/05/2013 04:34 AM Aaron Marcuse-Kubitza

translate: Changed dictionary to thesaurus, since the map used actually has synonyms rather than definitions

7447 02/05/2013 04:31 AM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Translate the thesaurus's output terms using itself in order to map a synonym of an ambiguous term directly to its alternatives list rather than only to the ambiguous term itself

7446 02/05/2013 04:26 AM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Run collapse_multimap on the generated map so that all alternatives are included, rather than just the first alternative, when translate maps an ambiguous term

7445 02/05/2013 04:25 AM Aaron Marcuse-Kubitza

redmine_synonyms: Fixed bug where need to output a CSV rather than TSV to be usable by other programs that use map spreadsheets

7444 02/05/2013 04:23 AM Aaron Marcuse-Kubitza

Added collapse_multimap, which collapses multimap entries in a spreadsheet dictionary

7443 02/05/2013 03:45 AM Aaron Marcuse-Kubitza

mappings/Veg+-VegCore.csv: Separate alternatives of ambiguous terms with , instead of ", " for easier machine-parsability

7442 02/05/2013 03:31 AM Aaron Marcuse-Kubitza

redmine_synonyms: Added support for ambiguous terms, which unlike the synonyms format nests the term (the alternative) under the synonym (the ambiguous term) rather than the synonym under the term. Note that ambiguous terms must also be prefixed with ? to differentiate them from composites (e.g. recordedBy_givenName), which use the same _-based naming convention.

7441 02/05/2013 03:08 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7440 02/05/2013 02:49 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7439 02/05/2013 02:22 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: Renamed scientificNameWithMorphospecies to taxonNameWithMorphospecies because it does not contain the scientific name author, as required by DwC scientificName <http://rs.tdwg.org/dwc/terms/#scientificName>

7438 02/05/2013 01:56 AM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.tables.csv: Exclude ambiguous table names, which should not be part of the tables summary (as neither are table synonyms)

7437 02/05/2013 01:51 AM Aaron Marcuse-Kubitza

input.Makefile: $(translate?): Merged with $(translate), which is not used independently

7436 02/05/2013 01:50 AM Aaron Marcuse-Kubitza

input.Makefile: Use new translate_ci instead of translate

7435 02/05/2013 01:47 AM Aaron Marcuse-Kubitza

mappings/Makefile: Use new translate_ci instead of translate

7434 02/05/2013 01:39 AM Aaron Marcuse-Kubitza

Added translate_ci

7433 02/04/2013 11:03 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: institutionCode list->sourcename mapping: _split(): Also match ; as a separator, and match separators with or without a following space

7432 02/02/2013 05:39 PM Aaron Marcuse-Kubitza

mappings/Makefile: Added target to create Veg+-VegCore.csv from VegCore.htm, initially commented out until all the synonyms in the existing Veg+-VegCore.csv are added to the VegCore data dictionary <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore_data_dictionary>

7431 02/02/2013 05:38 PM Aaron Marcuse-Kubitza

Added redmine_synonyms, which translates a Redmine HTML page to a thesaurus

7430 02/02/2013 04:37 PM Aaron Marcuse-Kubitza

lockfile: Linux: Documented why newgrp and recursive invocation of lockfile are needed

7429 02/02/2013 04:33 PM Aaron Marcuse-Kubitza

lockfile: Linux: Fixed bug where need to change primary group of the dotlockfile process to the group of the dir to contain the lockfile, because dotlockfile otherwise reports a "permission denied" error (even though the directory is actually writable, dotlockfile thinks it isn't). Running dotlockfile with a different primary group is complicated because newgrp, the command that does this, does not pass arguments to the new process, so they must instead be passed via environment variables and a recursive invocation of lockfile (with the $inner recursion flag set). Additionally, exec cannot be used to propagate the PPID (needed by dotlockfile) because newgrp creates a new process rather than using exec, so it must be manually entered into the lockfile after dotlockfile runs.

7428 02/02/2013 02:41 PM Aaron Marcuse-Kubitza

lockfile: Linux: Fixed bug where need to lower retry count to avoid overflowing the retries variable

7427 02/02/2013 02:37 PM Aaron Marcuse-Kubitza

lockfile: Linux: Added workaround for bug in dotlockfile where using -1 to retry indefinitely doesn't work, so need to use large integer instead

7426 02/02/2013 01:49 PM Aaron Marcuse-Kubitza

lockfile: Linux: Use bin/dotlockfile instead of the system's dotlockfile, because the system's dotlockfile is SETGID mail, which prevents it from creating lockfiles in a directory owned by the bien user and group when being run by the login user

7425 02/02/2013 01:38 PM Aaron Marcuse-Kubitza

bin/: svn:ignore: Added dotlockfile, which is copied from the system during installation

7424 02/02/2013 01:30 PM Aaron Marcuse-Kubitza

bin/: svn:ignore: Removed no longer applicable test_output

7423 02/02/2013 01:26 PM Aaron Marcuse-Kubitza

root Makefile: misc-Linux: Added command to copy dotlockfile to the bin/ dir, so that it can be used without being SETGID mail, which would prevent it from creating lockfiles in a directory owned by the bien user and group when being run by the user

7422 02/02/2013 01:24 PM Aaron Marcuse-Kubitza

root Makefile: core: Added misc-* to install other dependencies

7421 02/02/2013 11:56 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: scientificNameWithMorphospecies: Removed no longer needed canon_taxonverbatim.family alternative, since the family will be included in the canon_taxonlabel.taxonomicname by the mappings

7420 02/02/2013 11:49 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: scientificNameWithMorphospecies: Fixed bug where need to use canon_*taxonlabel*.taxonomicname instead of canon_taxonverbatim.taxonomicname as one of the alternatives because only canon_taxonlabel.taxonomicname is guaranteed to be populated by the mappings, while canon_taxonverbatim.taxonomicname will only be populated if the datasource explicitly specifies that field. This distinction is only meaningful for data without a TNRS match, as TNRS supplies canon_taxonverbatim.taxonomicname.

7419 02/02/2013 11:28 AM Aaron Marcuse-Kubitza

import_all: after_import(): Added wait on tnrs.make's lockfile to ensure that all background scrubbing processes are complete before creating the analytical DB

7418 02/02/2013 11:18 AM Aaron Marcuse-Kubitza

import_all: Moved `waitpid $jobs` into after_import()

7417 02/01/2013 04:57 PM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: Fixed table sizes

7416 02/01/2013 04:51 PM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: Regenerated exports

7415 02/01/2013 04:34 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: removed all accessioncode fields, as VegBIEN does not use them

7414 02/01/2013 03:10 PM Aaron Marcuse-Kubitza

Added inputs/FIA/_src/FIADB_version4.accdb and FIADB_version4.sql (created from it using Access To PostgreSQL and the additional transformations at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Tools#MS-Access-database-MDB&gt;)

7413 01/31/2013 08:20 PM Aaron Marcuse-Kubitza

Added inputs/FIA/COND_unique/, generated from new FIA data

7412 01/31/2013 08:05 PM Aaron Marcuse-Kubitza

inputs/FIA/FIA_COND_unique/create.sql: Fixed bug where need to remove `CREATE TABLE :table AS` at beginning because that is added by the make target

7411 01/31/2013 08:03 PM Aaron Marcuse-Kubitza

inputs/FIA/geoscrub.~.clean_up.sql: Moved creation of FIA_COND_unique to FIA_COND_unique/create.sql

7410 01/31/2013 07:40 PM Aaron Marcuse-Kubitza

README.TXT: Full database import: Updated time until import_all returns control to the shell to account for the TNRS names now being imported concurrently with the inputs rather than before them

7409 01/31/2013 07:31 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Also include morphospecies in the accepted taxondetermination's taxonverbatim, so that it can easily be retrieved by the analytical DB views

7408 01/31/2013 07:15 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: scientificNameWithMorphospecies: Fixed bug where need to use the taxonName or scientificName when the name components are not provided, as is the case when there is no scrubbed taxondetermination (because TNRS returns no match)

7407 01/31/2013 06:08 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. This adds Brad's DwC ID terms and their definitions in <https://projects.nceas.ucsb.edu/nceas/attachments/download/621/vegbien_identifier_examples.xlsx&gt;.

7406 01/31/2013 05:06 PM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: Regenerated exports

7405 01/31/2013 04:04 PM Aaron Marcuse-Kubitza

join: Added support for direct mappings to VegBIEN by passing through outputs that start with / (indicating an XPath rather than a term)