Project

General

Profile

Statistics
| Revision:

# Date Author Comment
7545 02/14/2013 02:51 AM Aaron Marcuse-Kubitza

inputs/ARIZ/: Added SQL export for refresh

7544 02/14/2013 02:33 AM Aaron Marcuse-Kubitza

my2pg.data: Translate indefinite (zero) months which have a definite day. This is unusual, but does appear in some data such as the ARIZ DB.

7543 02/14/2013 02:28 AM Aaron Marcuse-Kubitza

my2pg.data: Translate indefinite dates (dates with 0 as the month or day)

7542 02/14/2013 02:23 AM Aaron Marcuse-Kubitza

my2pg: Use my2pg.data to perform data-only replacements, instead of duplicating them in both my2pg and my2pg.data

7541 02/14/2013 02:01 AM Aaron Marcuse-Kubitza

my2pg: named UNIQUE KEYs: Comment out the name because PostgreSQL requires it to be globally unique, but MySQL only requires it to be unique within the table

7540 02/14/2013 01:53 AM Aaron Marcuse-Kubitza

my2pg: Translate UNIQUE KEYs instead of removing them

7539 02/14/2013 01:49 AM Aaron Marcuse-Kubitza

my2pg*: Removed KEYs: Comment out the definition rather than removing it

7538 02/14/2013 01:45 AM Aaron Marcuse-Kubitza

my2pg*: Remove FOREIGN KEYs because MySQL does not dump tables in dependency order, which prevents PostgreSQL from creating tables whose fkeys refer to a later table

7537 02/14/2013 01:33 AM Aaron Marcuse-Kubitza

my2pg*: Replacing invalid table elements to remove them: Use a dummy CHECK constraint instead of a boolean field to avoid adding fields to the table. The elements can't always simply be removed because sed can't remove the trailing comma of the previous element, and removing the following comma doesn't work for the last element in the table.

7536 02/14/2013 12:11 AM Aaron Marcuse-Kubitza

my2pg*: Replace '0000-00-00 00:00:00' with '-infinity'

7535 02/14/2013 12:04 AM Aaron Marcuse-Kubitza

my2pg: Replace datetime with timestamp

7534 02/13/2013 11:59 PM Aaron Marcuse-Kubitza

my2pg: Remove COLLATE field attribute

7533 02/13/2013 11:56 PM Aaron Marcuse-Kubitza

lib/MySQL.*.sql.make: Documented that $server user/host are for ssh, not the DB

7532 02/13/2013 11:55 PM Aaron Marcuse-Kubitza

lib/MySQL.*.sql.make: Documented that $server can also contain a username (which will be used by ssh)

7531 02/13/2013 11:51 PM Aaron Marcuse-Kubitza

my2pg_export: Use the --quick option to facilitate exporting large tables (it avoids retrieving all rows before outputting any of them)

7530 02/13/2013 11:00 PM Aaron Marcuse-Kubitza

README.TXT: Datasource setup: Added instructions for MS Access databases

7529 02/13/2013 10:43 PM Aaron Marcuse-Kubitza

README.TXT: Datasource setup: MySQL inputs: Added instruction to skip the Add input data for each table section

7528 02/13/2013 10:40 PM Aaron Marcuse-Kubitza

inputs/NY/: Added SQL export for refresh

7527 02/12/2013 01:08 PM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. Brad's new DwC ID terms spreadsheet has now been added, and a number of the ID terms clarified, disambiguated, and recategorized. In particular, institutionCode has now been split into the custodialInstitutions and collectingInstitution, to differentiate between which institution has the specimen vs. stamped the specimen. This distinction is important because the catalogNumber, stamped on the specimen, is only unique within the collectingInstitution. Most datasources don't unambiguously specify which institution their institutionCode is referring to, so it has been assumed to be custodialInstitutions unless a data dictionary says otherwise (as is the case for UNCC). In addition, a MatchedTaxonDetermination table has been added with the *_matched fields from TNRS.

7526 02/12/2013 12:15 PM Aaron Marcuse-Kubitza

inputs/CVS/observation_/map.csv: baseSaturation: Resolved ambiguous term

7525 02/12/2013 12:09 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.vocab.csv: Ignore leading ? when sorting so that ambiguous terms sort alphabetically with other terms. This prevents terms from moving from their previous location when they become ambiguous.

7524 02/12/2013 12:07 PM Aaron Marcuse-Kubitza

Added sort_ci to sort a spreadsheet, ignoring leading punctuation

7523 02/12/2013 12:05 PM Aaron Marcuse-Kubitza

mappings/VegCore.vocab.csv: Changed line endings to \r\n in preparation for having a Python script run on it (which changes the line endings)

7522 02/12/2013 11:47 AM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.vocab.csv: Added back ambiguous terms, so that the vocabulary contains all terms defined by VegCore, regardless of whether they are ambiguous or unambiguous terms

7521 02/12/2013 11:44 AM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.vocab.csv: Added back synonyms, so that the vocabulary contains all terms defined by VegCore, regardless of whether they are synonyms or primary terms. This also prevents VegCore.vocab.csv from losing entries when terms are renamed, which made it difficult to verify that no terms were lost when refactoring.

7520 02/12/2013 05:50 AM Aaron Marcuse-Kubitza

inputs/MO/Specimen/postprocess.sql: Remove frameshifted rows by detecting InstitutionCodes without any letters

7519 02/12/2013 04:59 AM Aaron Marcuse-Kubitza

inputs/ARIZ/Specimen/map.csv: CollectorNumber/FieldNumber: Use /_first to map these identical fields to the same location

7518 02/12/2013 04:54 AM Aaron Marcuse-Kubitza

inputs/ARIZ/Specimen/map.csv: Fixed bug where the column names for InstitutionCode and CollectionCode were reversed in the source data

7517 02/12/2013 04:14 AM Aaron Marcuse-Kubitza

inputs/*/Specimen/map.csv for Canadensys sources: Remapped institutionID to UNUSED

7516 02/09/2013 07:45 AM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki. The original*, accepted*, and verbatim* Taxon fields have now been moved to separate OriginalTaxonDetermination, AcceptedTaxonDetermination, and TaxonVerbatim tables.

7515 02/09/2013 06:52 AM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki

7514 02/09/2013 06:34 AM Aaron Marcuse-Kubitza

mappings/VegCore.htm: Regenerated from wiki

7513 02/09/2013 04:08 AM Aaron Marcuse-Kubitza

README.TXT: Maintenance: VegCore data dictionary: Replaced VegCore.*.csv with VegCore.htm because now that VegCore.*.csv are sorted alphabetically, they generally don't change when VegCore.htm changes

7512 02/09/2013 04:04 AM Aaron Marcuse-Kubitza

mappings/VegCore.*.csv: Regenerated from wiki. A plain text label is now used for Replace with, which fixes a bug where the PRIVATE permalink pointed to its Replace with in realLatitude instead of its definition.

7511 02/09/2013 03:55 AM Aaron Marcuse-Kubitza

redmine_synonyms: Support plain text labels other than Alternative, such as Replace with

7510 02/09/2013 03:13 AM Aaron Marcuse-Kubitza

mappings/VegCore.*.csv: Regenerated from wiki. Alternatives now contain the "Alternative" label as plain text rather than as an image title, thus avoiding an HTML anchor conflict with the definition and allowing ambiguous terms to be placed before their alternatives as well as after.

7509 02/09/2013 03:11 AM Aaron Marcuse-Kubitza

README.TXT: Maintenance: VegCore data dictionary: Updated VegCore.csv filename to VegCore.*.csv

7508 02/09/2013 02:57 AM Aaron Marcuse-Kubitza

redmine_synonyms: Support alternatives which contain the "Alternative" label as plain text rather than as an image title. This is done to include the "Alternative" label in the HTML anchor and thus prevent the anchor from conflicting with the actual definition of the alternative (which would otherwise have the same anchor text). This allows ambiguous terms to be placed before their alternatives as well as after, because there won't be anchor conflicts that need to be resolved with careful ordering.

7507 02/09/2013 02:48 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. Taxon terms with prefixes for other TaxonDeterminations now indicate the analogous term in an "analogous to" label next to the term

7506 02/09/2013 02:47 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. Taxon terms with prefixes for other TaxonDeterminations now indicate the analogous term in an "analogous to" label next to the term

7505 02/09/2013 02:47 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. Taxon terms with prefixes for other TaxonDeterminations now indicate the analogous term in an "analogous to" label next to the term

7504 02/07/2013 01:57 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: datasourceRecordID: Fixed bug where also need to add datasourceRecordID next to occurrenceID for an institutionCode remap switch

7503 02/07/2013 01:57 PM Aaron Marcuse-Kubitza

inputs/bien_web/observation/test.xml.ref: Regenerated

7502 02/07/2013 01:48 PM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times using the import_times bugfix for times longer than a day

7501 02/07/2013 01:45 PM Aaron Marcuse-Kubitza

import_times: times(): Fixed bug where need to match whitespace in times, in order to match times with days

7500 02/07/2013 12:00 PM Aaron Marcuse-Kubitza

inputs/*/Specimen/map.csv: Remapped ID to datasourceRecordID

7499 02/07/2013 11:55 AM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped datasourceRecordID

7498 02/07/2013 11:51 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

7497 02/07/2013 08:38 AM Aaron Marcuse-Kubitza

inputs/FIA/_src/_README.TXT: Documented that the refresh is missing some PLT_CN values present in the original version

7496 02/07/2013 08:33 AM Aaron Marcuse-Kubitza

inputs/FIA/import_order.txt: Reverted back to using FIA_COND_unique instead of COND_unique because the PLT_CN IDs in the refresh don't match the PLT_CN IDs in the original version, making COND_unique and Organism incompatible

7495 02/07/2013 08:27 AM Aaron Marcuse-Kubitza

inputs/FIA/import_order.txt: Removed FIA_COND_unique, which is superseded by COND_unique

7494 02/07/2013 08:26 AM Aaron Marcuse-Kubitza

inputs/FIA/import_order.txt: Fixed bug where need to import COND_unique before Organism because the plot entries need to be created before they can be linked to by organisms

7493 02/07/2013 07:25 AM Aaron Marcuse-Kubitza

redmine_synonyms: sed pattern: Match <h# directly at the beginning of the line rather than after ^.*, which greatly speeds up the pattern matching because the first character is a literal character. (If <h# were not located at the left margin, the ^.* would unfortunately still be needed because the beginning of the line needs to be matched in order to be removed by the replacement operation.)

7492 02/07/2013 07:22 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. Alternatives are now able to use h3 instead of h4 (which had display problems). realLatitude/Longitude is now no longer needs the ? prefix to have its replacement (PRIVATE) interpreted as an alternative, and thus is properly able to be included in the vocabulary.

7491 02/07/2013 07:16 AM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.vocab.csv: Use the term's type label instead of its header level to determine if it's a synonym or alternative. This allows header levels to be chosen for presentational reasons rather than being constrained by being parsable.

7490 02/07/2013 07:05 AM Aaron Marcuse-Kubitza

redmine_synonyms: Don't require ambiguous terms to start with ?, because the ambiguous term for an alternative can be identified simply by choosing the last term that didn't have a type label (previously, this would have been the last term that wasn't h3 or h4)

7489 02/07/2013 07:01 AM Aaron Marcuse-Kubitza

redmine_synonyms: Use the term's type label instead of its header level to determine if it's a synonym or alternative. This allows header levels to be chosen for presentational reasons rather than being constrained by being parsable.

7488 02/07/2013 06:26 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. The data dictionary has been reformatted to be much more vertically compact, by placing the term type (Synonym, Alternative, etc.) and sources (From:) on the same line as the term. Note that globalUniqueIdentifier_SpeciesLink has been removed from the vocabulary because a definition entry has been added for it (when this entry is missing, the term is incorrectly identified as a primary term).

7487 02/07/2013 06:21 AM Aaron Marcuse-Kubitza

mappings/Makefile, redmine_synonyms: Updated for new VegCore data dictionary format, which prefixes the term type (Synonym, Alternative, etc.) to the term instead of including it as a section label. This ensures that the term type of a non-primary term is shown next to the term when it is visited via a permalink, which causes the term header to appear at the top of the screen and obscures the section header containing the type.

7486 02/07/2013 06:00 AM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.thesaurus.csv: removal of tables: ignore errors if grep found no match

7485 02/07/2013 02:06 AM Aaron Marcuse-Kubitza

Renamed mappings/VegCore.csv to VegCore.vocab.csv and Veg+-VegCore.csv to VegCore.thesaurus.csv for clarity

7484 02/07/2013 02:03 AM Aaron Marcuse-Kubitza

mappings/Makefile, input.Makefile: Renamed $(dict) to $(thesaurus) because Veg+-VegCore.csv is actually a thesaurus, not a dictionary

7483 02/07/2013 01:57 AM Aaron Marcuse-Kubitza

mappings/Makefile: Replaced occurrences of VegCore.csv with $(vocab) and Veg+-VegCore.csv with $(dict)

7482 02/06/2013 07:34 PM Aaron Marcuse-Kubitza

README.TXT: Maintenance: VegCore data dictionary: When moving terms, check that no terms were lost: Updated steps now that VegCore.csv and Veg+-VegCore.csv are sorted by name, so that a comparison of added/deleted counts is not necessary and a simple `svn di` can be used

7481 02/06/2013 07:33 PM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Sort terms by name so that reordering terms in the VegCore data dictionary does not cause Veg+-VegCore.csv to change. This makes it much easier to identify synonyms and ambiguous terms that were accidentally deleted during a data dictionary refactoring. (Note that these are no longer included in VegCore.csv, so this is required in addition to sorting VegCore.csv by name.)

7480 02/06/2013 07:26 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Sort terms by name so that reordering terms in the VegCore data dictionary does not cause VegCore.csv to change. This makes it much easier to identify terms that were accidentally deleted during a data dictionary refactoring.

7479 02/05/2013 06:19 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. This adds cf_aff.

7478 02/05/2013 06:18 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Filter out namespaces by matching only terms whose header links within the data dictionary

7477 02/05/2013 06:08 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. This causes TNRS's Annotations (cf/aff) to be mapped into VegBIEN.

7476 02/05/2013 06:05 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: matched*Fit_fraction: Remapped to taxonconfidence instead of taxonfit

7475 02/05/2013 05:56 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Fixed bug where need to remove duplicates, which are no longer supported by canon, by removing alternatives of ambiguous terms when these occur separately from their definitions

7474 02/05/2013 05:29 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Removed synonyms and ambiguous terms, since the canonicalization of them is handled by Veg+-VegCore.csv. This also reduces the time it takes canon to build the in-memory Python dict of replacements, which scales to all inputs and should speed up the build/test command.

7473 02/05/2013 05:22 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Removed synonyms, since the canonicalization of them is handled by Veg+-VegCore.csv

7472 02/05/2013 05:10 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Match terms by header # instead of matching all anchors, in order to include the leading ? before an ambiguous term

7471 02/05/2013 04:42 PM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Generate dynamically from VegCore.htm, which allows the VegCore thesaurus to be automatically kept up to date. More importantly, it allows terms in all map spreadsheets to be updated simultaneously when a term is renamed (e.g. by replacing a term with one of its synonyms).

7470 02/05/2013 04:40 PM Aaron Marcuse-Kubitza

mappings/VegX-VegCore.csv: Applied term renamings from the new dynamically generated Veg+-VegCore.csv. Updates to VegCore term names that have occurred since the data dictionary was created are now able to take effect, which involves remapping several fields.

7469 02/05/2013 04:32 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv, inputs/*/*/map.csv: Applied term renamings from the new dynamically generated Veg+-VegCore.csv, which reflects the current state of the data dictionary. (Permanently switching to the new Veg+-VegCore.csv will be a separate change.) Updates to VegCore term names that have occurred since the data dictionary was created are now able to take effect, which involves remapping and inferring units on several fields.

7468 02/05/2013 04:27 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped basalDiameter_in

7467 02/05/2013 04:15 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped diameterBreastHeightGentry_cm, basalDiameter_cm, precipitation_mm

7466 02/05/2013 04:14 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Added _mm_to_m()

7465 02/05/2013 03:56 PM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Fixed bugs where also need to filter out ambiguous tables, but shouldn't filter out acronyms (which are regular fields)

7464 02/05/2013 03:40 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: locationID->location.sourceaccessioncode: Removed restriction that this mapping can't occur if geovalidation information is present. The locationID is no longer mapped to the place.sourceaccessioncode, so this filter is not necessary.

7463 02/05/2013 03:38 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

7462 02/05/2013 03:19 PM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Fixed bug where need to filter out table names to avoid applying table replacements to fields which have the same name as a table

7461 02/05/2013 03:03 PM Aaron Marcuse-Kubitza

inputs/Madidi/map.csv: Fixed bug where needed to remove duplicate input names, now that translate doesn't allow them

7460 02/05/2013 01:59 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegX-VegCore.csv: Sort by the input column instead of the output column to keep the sort order stable across VegCore term renames

7459 02/05/2013 01:46 PM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Before running collapse_multimap, canonicalize alternatives of ambiguous terms using unambiguous mappings. This ensures that the alternatives lists contain only canonical VegCore terms rather than synonyms.

7458 02/05/2013 01:43 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. All synonyms are now hyperlinked, allowing them to be matched by redmine_synonyms.

7457 02/05/2013 01:31 PM Aaron Marcuse-Kubitza

mappings/Veg+-VegCore.csv: Removed Sources, Definition columns because source information is now in the VegCore data dictionary

7456 02/05/2013 01:25 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. Ambiguous terms newly available to redmine_synonyms due to the bugfix now have multiple alternatives.

7455 02/05/2013 01:25 PM Aaron Marcuse-Kubitza

redmine_synonyms: Ambiguous terms: Fixed bug where need to use header # instead of term name to determine whether a term is an alternative, because some alternatives (e.g. verbatimElevation) don't follow the units-suffix naming convention.

7454 02/05/2013 12:58 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. All ambiguous terms now have multiple alternatives, preventing them from being automapped to a single alternative without prompting the user for confirmation

7453 02/05/2013 12:50 PM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: translate: Fixed bug where need to run on $@ instead of $<

7452 02/05/2013 12:49 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. All ambiguous terms now have multiple alternatives, preventing them from being automapped to a single alternative without prompting the user for confirmation

7451 02/05/2013 12:22 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. All mappings/Veg+-VegCore.csv terms are now added as synonyms or separate terms.

7450 02/05/2013 10:26 AM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki. Most ambiguous terms are now split into alternatives, and most mappings/Veg+-VegCore.csv terms are now added as synonyms.

7449 02/05/2013 06:12 AM Aaron Marcuse-Kubitza

canon: Raise an error if two input terms map to the same simplified string

7448 02/05/2013 04:34 AM Aaron Marcuse-Kubitza

translate: Changed dictionary to thesaurus, since the map used actually has synonyms rather than definitions

7447 02/05/2013 04:31 AM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Translate the thesaurus's output terms using itself in order to map a synonym of an ambiguous term directly to its alternatives list rather than only to the ambiguous term itself

7446 02/05/2013 04:26 AM Aaron Marcuse-Kubitza

mappings/Makefile: Veg+-VegCore.csv: Run collapse_multimap on the generated map so that all alternatives are included, rather than just the first alternative, when translate maps an ambiguous term