Project

General

Profile

Activity

From 01/23/2013 to 02/21/2013

02/21/2013

05:13 PM Revision 7644: schemas/vegbien.sql: analytical_stem_view: scientificName_verbatim: Use the taxonname (which omits the family and author) instead when available, at Brad's request
Aaron Marcuse-Kubitza
04:02 PM Revision 7643: schemas/vegbien.sql: Disambiguated taxonlabel.taxonomicname, taxonverbatim.taxonomicname, and taxonverbatim.taxonname
Aaron Marcuse-Kubitza
03:14 PM Revision 7642: schemas/vegbien.sql: taxon_trait_view: Use taxonverbatim.taxonname (the binomial) in preference to taxonverbatim.taxonomicname (the binomial+author) or taxonlabel.taxonomicname (the family+binomial+author), at Mark's request
Aaron Marcuse-Kubitza
02:05 PM Revision 7641: to_do/timeline.2013.xls: Obtain and load any additional new data: Extended beyond April because it's ongoing
Aaron Marcuse-Kubitza
02:02 PM Revision 7640: README.TXT: Full database import: On nimoy: Moved analytical_aggregate table creation right after analytical_aggregate..csv download because it can happen while the file is downloaded
Aaron Marcuse-Kubitza
11:13 AM Revision 7639: to_do/timeline.2013.xls: Reload existing data in need of refresh: Added additional cell because this is still in progress
Aaron Marcuse-Kubitza
11:11 AM Revision 7638: to_do/timeline.2013.xls: Placed check mark in completed cells. Multi-cell tasks that are partially complete have check marks in part of their cells to indicate this.
Aaron Marcuse-Kubitza

02/20/2013

05:35 PM Revision 7637: Added inputs/FIA/REF_*/
Aaron Marcuse-Kubitza
05:33 PM Revision 7636: inputs/FIA/_src/Makefile: Table bundling: Also unpack FIADB_REFERENCE/
Aaron Marcuse-Kubitza
05:13 PM Revision 7635: Added inputs/FIA/_src/FIADB_REFERENCE.ZIP.md5, .url
Aaron Marcuse-Kubitza
04:11 PM Revision 7634: schemas/vegbien.sql: provider_count_view: Group data by dataprovider and dataset, rather than by each separately, to satisfy the GBIF attribution requirements as described by Brad
Aaron Marcuse-Kubitza
02:00 PM Revision 7633: schemas/vegbien.sql: analytical_plot/specimen: Moved threatened_bien after growthForm as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#SpeciesLink>
Aaron Marcuse-Kubitza
01:54 PM Revision 7632: schemas/vegbien.sql: analytical_stem_view: Moved threatened_bien after growthForm as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#SpeciesLink>
Aaron Marcuse-Kubitza
01:53 PM Revision 7631: schemas/vegbien.sql: sync_analytical_stem_to_view(): Also drop and re-create new dependent views analytical_plot, analytical_specimen
Aaron Marcuse-Kubitza
01:20 PM Revision 7630: inputs/TEAM/V*/map.csv: Mapped plotArea_ha based on "1ha Plot Number" field name
Aaron Marcuse-Kubitza
01:12 PM Revision 7629: inputs/TEAM/V*/map.csv: Id: Remapped to individualObservationID
Aaron Marcuse-Kubitza
01:10 PM Revision 7628: mappings/VegCore.htm: Regenerated from wiki. verbatimScientificName, verbatimTaxonName have been merged into concatenatedScientificName, and ID has been disambiguated.
Aaron Marcuse-Kubitza
01:09 PM Revision 7627: inputs/*/*/map.csv: Automapped taxonConfidence
Aaron Marcuse-Kubitza
12:28 PM Revision 7626: schemas/vegbien.sql: analytical_stem_view: plotArea_ha: Fixed bug where need to use parent plot's area when subplot's area is not provided
Aaron Marcuse-Kubitza
12:18 PM Revision 7625: inputs/SALVIAS/plotMetadata_/create.sql: Added primary key
Aaron Marcuse-Kubitza
11:42 AM Revision 7624: inputs/NY/NY_DATA_20130212.~.utils.sql: pkey: Made table an enum instead of a text field so that Ecatalog_LatLong could be sorted before Ecatalog
Aaron Marcuse-Kubitza
11:32 AM Revision 7623: inputs/NY/Ecatalog_all/create.sql: primary key: Combine table, key into one value so that the primary key has a single column. This is necessary because multi-column primary keys are not supported by the import algorithm.
Aaron Marcuse-Kubitza
11:28 AM Revision 7622: inputs/NY/NY_DATA_20130212.~.utils.sql: Added pkey record type for use by Ecatalog_all's generated pkey
Aaron Marcuse-Kubitza
10:47 AM Revision 7621: inputs/NY/Ecatalog_all/create.sql: Add table column which scopes the key field. Add primary key to allow sorting by it. inputs/NY/Ecatalog_all/map.csv: Documented that UniqueNYInternalRecordNumber is sometimes duplicated.
Aaron Marcuse-Kubitza
09:20 AM Revision 7620: make_analytical_db: Wait for tnrs.make to acquire the tnrs.make lock if being run right after import
Aaron Marcuse-Kubitza
09:13 AM Revision 7619: schemas/vegbien.sql: analytical_stem_view: taxonName_matched: Fixed bug where need to use the matched_taxonverbatim instead of the datasource_taxonverbatim
Aaron Marcuse-Kubitza
08:58 AM Revision 7618: Moved wait on tnrs.make lock from import_all to make_analytical_db, so that running make_analytical_db for a one-time import also waits on the lock
Aaron Marcuse-Kubitza
08:56 AM Revision 7617: schemas/vegbien.sql: taxondetermination: taxondetermination_unique: Added determinationtype so that when the matched and accepted determinations are the same, they still both get created rather than the second one being removed due to the unique constraint
Aaron Marcuse-Kubitza
07:59 AM Revision 7616: schemas/vegbien.sql: analytical_specimen: Removed speciesBinomialWithMorphospecies because it doesn't apply to specimens
Aaron Marcuse-Kubitza
07:58 AM Revision 7615: schemas/vegbien.sql: Added analytical_plot view
Aaron Marcuse-Kubitza
07:55 AM Revision 7614: schemas/vegbien.sql: Added analytical_specimen view
Aaron Marcuse-Kubitza
07:44 AM Revision 7613: schemas/vegbien.sql: analytical_stem_view: Moved recordedBy, recordNumber before dateCollected as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#ACAD>
Aaron Marcuse-Kubitza
07:40 AM Revision 7612: schemas/vegbien.ERD.mwb: Synced with schema
Aaron Marcuse-Kubitza
07:38 AM Revision 7611: schemas/vegbien.sql: Added reproductiveCondition
Aaron Marcuse-Kubitza
07:33 AM Revision 7610: mappings/VegCore-VegBIEN.csv: Mapped reproductiveCondition
Aaron Marcuse-Kubitza
07:28 AM Revision 7609: schemas/vegbien.sql: plantobservation: Added reproductivecondition
Aaron Marcuse-Kubitza
05:33 AM Revision 7608: mappings/VegCore.htm: Regenerated from wiki. matched*Fit_fraction has been renamed to matched*Confidence_fraction.
Aaron Marcuse-Kubitza
05:32 AM Revision 7607: inputs/.TNRS/public.unscrubbed_taxondetermination_view/map.csv: Updated for new mappings/VegCore.htm
Aaron Marcuse-Kubitza
05:10 AM Revision 7606: inputs/bien_web/observation/map.csv: Re-automapped taxonMorphospecies
Aaron Marcuse-Kubitza
05:08 AM Revision 7605: mappings/VegCore.htm: Regenerated from wiki. Data owner terms and taxon synonyms have been added, and morphospecies has been disambiguated.
Aaron Marcuse-Kubitza
04:51 AM Revision 7604: schemas/vegbien.sql: analytical_stem_view: Moved identifiedBy, dateIdentified, identificationRemarks right after the *_verbatim terms that they relate to, as requested by Brad <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#ACAD>
Aaron Marcuse-Kubitza
02:25 AM Revision 7603: schemas/vegbien.sql: analytical_stem_view: Use new concat_delim() instead of array_to_string() surrounded by NULLIF(..., '')
Aaron Marcuse-Kubitza
02:19 AM Revision 7602: schemas/vegbien.sql: Added concat_delim()
Aaron Marcuse-Kubitza
01:43 AM Revision 7601: schemas/vegbien.sql: analytical_stem_view: Removed deprecated taxonNameWithMorphospecies now that we have speciesBinomialWithMorphospecies
Aaron Marcuse-Kubitza
01:17 AM Revision 7600: schemas/vegbien.sql: analytical_stem_view: speciesBinomial: Added morphospecies suffix to create speciesBinomialWithMorphospecies
Aaron Marcuse-Kubitza
12:15 AM Revision 7599: inputs/import.stats.xls: Updated import times
Aaron Marcuse-Kubitza
12:06 AM Revision 7598: README.TXT: Full database import: Check that unscrubbed_taxondetermination_view returns no rows: Documented that this takes 90 s with LIMIT 1
Aaron Marcuse-Kubitza

02/19/2013

11:16 PM Revision 7597: schemas/vegbien.sql: _taxon_family_require_std(): Also allow non-aceae families accepted by TNRS
Aaron Marcuse-Kubitza
09:38 PM Revision 7596: Added inputs/SALVIAS/_archive/salvias_plots.*.sql.zip.md5
Aaron Marcuse-Kubitza
09:35 PM Revision 7595: Added inputs/VegBank/_archive/vegbank_for_bien.tar.gz.url
Aaron Marcuse-Kubitza
09:29 PM Revision 7594: Added inputs/U/UtrechtHerbarium.csv.tar.gz.url
Aaron Marcuse-Kubitza
09:28 PM Revision 7593: Added inputs/TEAM/_archive/ci-team_extract.tar.gz.url
Aaron Marcuse-Kubitza
09:27 PM Revision 7592: Added inputs/SpeciesLink/_archive/specieslink*.txt.gz.url
Aaron Marcuse-Kubitza
09:22 PM Revision 7591: Added inputs/REMIB/_archive/remib_raw.csv.tar.gz.url
Aaron Marcuse-Kubitza
09:19 PM Revision 7590: Added inputs/NY/NYSpecimenDataAmericas.csv.tar.gz.url
Aaron Marcuse-Kubitza
09:17 PM Revision 7589: Added inputs/NCU/_archive/NCU-NCSC_2010-02-12.csv.tar.gz.url
Aaron Marcuse-Kubitza
09:14 PM Revision 7588: Added inputs/MO/mo_digirexport.tar.gz.url
Aaron Marcuse-Kubitza
09:13 PM Revision 7587: Added inputs/Madidi/_archive/2010-1-2/madidi_plots_original_12jan2010.zip.url
Aaron Marcuse-Kubitza
09:11 PM Revision 7586: Added inputs/GBIF/gbif_extract.tar.gz.url
Aaron Marcuse-Kubitza
09:10 PM Revision 7585: Added inputs/FIA/fia_extract.tar.gz.url
Aaron Marcuse-Kubitza
09:08 PM Revision 7584: Added inputs/CVS/_archive/CVS-allTaxonOccurrences_2010-01-12.txt.tar.gz.url
Aaron Marcuse-Kubitza
09:04 PM Revision 7583: Added inputs/ARIZ/ARIZ_DiGIR_21012010.csv.tar.gz.url
Aaron Marcuse-Kubitza
08:55 PM Revision 7582: Added inputs/UNCC/Specimen/UNCC.csv.url, UNCC.csv.md5
Aaron Marcuse-Kubitza
08:45 PM Revision 7581: Added inputs/XAL/_src/digir.xml.gz.md5
Aaron Marcuse-Kubitza
08:39 PM Revision 7580: Added inputs/UNCC/_src/ with UNCC.csv.zip.md5
Aaron Marcuse-Kubitza
08:23 PM Revision 7579: Added inputs/SpeciesLink/_src
Aaron Marcuse-Kubitza

02/16/2013

08:24 AM Revision 7578: README.TXT: Datasource setup: MySQL inputs: .sql exports: Use new mysql_bien to connect to the MySQL DB created for the datasource
Aaron Marcuse-Kubitza
08:22 AM Revision 7577: Added mysql_bien, which runs a MySQL command on the local MySQL server
Aaron Marcuse-Kubitza
08:06 AM Revision 7576: Added inputs/GBIF/_src/GBIFPortalDB-2012-12-11.dump.md5 (md5sum of the expanded file)
Aaron Marcuse-Kubitza
08:02 AM Revision 7575: root Makefile: MySQL: mysql-Linux: Also install phpMyAdmin
Aaron Marcuse-Kubitza
08:01 AM Revision 7574: root Makefile: MySQL: mysql-Linux: Split apt-get dependencies into separate commands, like for other apt-get commands, to avoid having one failed dependency prevent the following dependencies from being installed
Aaron Marcuse-Kubitza
07:57 AM Revision 7573: root Makefile: MySQL: *mysql_users: Also add bien_read user
Aaron Marcuse-Kubitza
07:49 AM Revision 7572: root Makefile: MySQL: Renamed *mysql_user to *mysql_users because there can be multiple users
Aaron Marcuse-Kubitza
06:51 AM Revision 7571: inputs/: Added .md5 files for all .zip, .gz
Aaron Marcuse-Kubitza
06:47 AM Revision 7570: Added inputs/HVAA/Specimen/Herbario_occur_1360871068.csv.url
Aaron Marcuse-Kubitza
06:39 AM Revision 7569: lib/common.Makefile: rsync: $(rsync*): Use --no-group because the file group is different depending on the machine
Aaron Marcuse-Kubitza
06:10 AM Revision 7568: input.Makefile: SVN: $(_svnFilesGlob): Also add .md5 files. This allows svn to track where unversioned files should be in the directory tree.
Aaron Marcuse-Kubitza
06:07 AM Revision 7567: input.Makefile: SVN: $(_svnFilesGlob): .url, .pdf, and README.TXT in the top-level dir: Fixed bug where had extra / after brace expr
Aaron Marcuse-Kubitza
06:00 AM Revision 7566: input.Makefile: SVN: $(_svnFilesGlob): Also add .url, .pdf, and README.TXT in the top-level dir
Aaron Marcuse-Kubitza
05:53 AM Revision 7565: input.Makefile: SVN: $(_svnFilesGlob): Add .url, .pdf, and README.TXT files in all subdirs, not just _src
Aaron Marcuse-Kubitza
05:25 AM Revision 7564: lib/common.Makefile: remote server: Use jupiter instead of vegbiendev, to ensure that all files get uploaded there rather than only to vegbiendev. This involves adding an extra database import step to download the uploaded files from jupiter onto vegbiendev.
Aaron Marcuse-Kubitza
02:50 AM Revision 7563: inputs/FIA/_src/Makefile: all: Extract zip files before running tables target, because it requires the created dirs
Aaron Marcuse-Kubitza
02:40 AM Revision 7562: schemas/vegbien.ERD.mwb: Fixed table sizes
Aaron Marcuse-Kubitza
01:17 AM Revision 7561: Removed no longer used fix_permissions. Use root fix_perms instead.
Aaron Marcuse-Kubitza
01:16 AM Revision 7560: Added root fix_perms
Aaron Marcuse-Kubitza

02/15/2013

11:58 PM Revision 7559: Moved Checksums from backups/Makefile to lib/common.Makefile so all dirs (including inputs/) can use md5sum testing
Aaron Marcuse-Kubitza
11:08 PM Revision 7558: lib/common.Makefile: $(remote): Made remote basepath configurable in $(remote_basepath)
Aaron Marcuse-Kubitza
11:04 PM Revision 7557: lib/common.Makefile: Renamed $(src_server) to $(remote_host) and $(src_user) to $(remote_user) for clarity
Aaron Marcuse-Kubitza
10:16 PM Revision 7556: inputs/GBIF/: Added refresh metadata
Aaron Marcuse-Kubitza

02/14/2013

11:49 AM Revision 7555: Added inputs/HVAA/
Aaron Marcuse-Kubitza
11:14 AM Revision 7554: Added inputs/ARIZ/_archive
Aaron Marcuse-Kubitza
11:13 AM Revision 7553: inputs/ARIZ/: Removed previous data now that it has been refreshed
Aaron Marcuse-Kubitza
11:08 AM Revision 7552: inputs/ARIZ/: Mapped refresh
Aaron Marcuse-Kubitza
09:48 AM Revision 7551: Added inputs/ARIZ/import_order.txt
Aaron Marcuse-Kubitza
09:22 AM Revision 7550: Added inputs/NY/_archive/
Aaron Marcuse-Kubitza
09:20 AM Revision 7549: inputs/NY/: Removed tables from previous extract
Aaron Marcuse-Kubitza
08:59 AM Revision 7548: inputs/NY/: Mapped refresh
Aaron Marcuse-Kubitza
08:58 AM Revision 7547: inputs/*/*/VegBIEN.csv: Regenerated from mappings/VegCore-VegBIEN.csv
Aaron Marcuse-Kubitza
08:52 AM Revision 7546: Added inputs/NY/import_order.txt
Aaron Marcuse-Kubitza
02:51 AM Revision 7545: inputs/ARIZ/: Added SQL export for refresh
Aaron Marcuse-Kubitza
02:33 AM Revision 7544: my2pg.data: Translate indefinite (zero) months which have a definite day. This is unusual, but does appear in some data such as the ARIZ DB.
Aaron Marcuse-Kubitza
02:28 AM Revision 7543: my2pg.data: Translate indefinite dates (dates with 0 as the month or day)
Aaron Marcuse-Kubitza
02:23 AM Revision 7542: my2pg: Use my2pg.data to perform data-only replacements, instead of duplicating them in both my2pg and my2pg.data
Aaron Marcuse-Kubitza
02:01 AM Revision 7541: my2pg: named UNIQUE KEYs: Comment out the name because PostgreSQL requires it to be globally unique, but MySQL only requires it to be unique within the table
Aaron Marcuse-Kubitza
01:53 AM Revision 7540: my2pg: Translate UNIQUE KEYs instead of removing them
Aaron Marcuse-Kubitza
01:49 AM Revision 7539: my2pg*: Removed KEYs: Comment out the definition rather than removing it
Aaron Marcuse-Kubitza
01:45 AM Revision 7538: my2pg*: Remove FOREIGN KEYs because MySQL does not dump tables in dependency order, which prevents PostgreSQL from creating tables whose fkeys refer to a later table
Aaron Marcuse-Kubitza
01:33 AM Revision 7537: my2pg*: Replacing invalid table elements to remove them: Use a dummy CHECK constraint instead of a boolean field to avoid adding fields to the table. The elements can't always simply be removed because sed can't remove the trailing comma of the previous element, and removing the following comma doesn't work for the last element in the table.
Aaron Marcuse-Kubitza
12:11 AM Revision 7536: my2pg*: Replace '0000-00-00 00:00:00' with '-infinity'
Aaron Marcuse-Kubitza
12:04 AM Revision 7535: my2pg: Replace datetime with timestamp
Aaron Marcuse-Kubitza

02/13/2013

11:59 PM Revision 7534: my2pg: Remove COLLATE field attribute
Aaron Marcuse-Kubitza
11:56 PM Revision 7533: lib/MySQL.*.sql.make: Documented that $server user/host are for ssh, not the DB
Aaron Marcuse-Kubitza
11:55 PM Revision 7532: lib/MySQL.*.sql.make: Documented that $server can also contain a username (which will be used by ssh)
Aaron Marcuse-Kubitza
11:51 PM Revision 7531: my2pg_export: Use the --quick option to facilitate exporting large tables (it avoids retrieving all rows before outputting any of them)
Aaron Marcuse-Kubitza
11:00 PM Revision 7530: README.TXT: Datasource setup: Added instructions for MS Access databases
Aaron Marcuse-Kubitza
10:43 PM Revision 7529: README.TXT: Datasource setup: MySQL inputs: Added instruction to skip the Add input data for each table section
Aaron Marcuse-Kubitza
10:40 PM Revision 7528: inputs/NY/: Added SQL export for refresh
Aaron Marcuse-Kubitza

02/12/2013

01:08 PM Revision 7527: mappings/VegCore.htm: Regenerated from wiki. Brad's new DwC ID terms spreadsheet has now been added, and a number of the ID terms clarified, disambiguated, and recategorized. In particular, institutionCode has now been split into the custodialInstitutions and collectingInstitution, to differentiate between which institution has the specimen vs. stamped the specimen. This distinction is important because the catalogNumber, stamped on the specimen, is only unique within the collectingInstitution. Most datasources don't unambiguously specify which institution their institutionCode is referring to, so it has been assumed to be custodialInstitutions unless a data dictionary says otherwise (as is the case for UNCC). In addition, a MatchedTaxonDetermination table has been added with the *_matched fields from TNRS.
Aaron Marcuse-Kubitza
12:15 PM Revision 7526: inputs/CVS/observation_/map.csv: baseSaturation: Resolved ambiguous term
Aaron Marcuse-Kubitza
12:09 PM Revision 7525: mappings/Makefile: VegCore.vocab.csv: Ignore leading ? when sorting so that ambiguous terms sort alphabetically with other terms. This prevents terms from moving from their previous location when they become ambiguous.
Aaron Marcuse-Kubitza
12:07 PM Revision 7524: Added sort_ci to sort a spreadsheet, ignoring leading punctuation
Aaron Marcuse-Kubitza
12:05 PM Revision 7523: mappings/VegCore.vocab.csv: Changed line endings to \r\n in preparation for having a Python script run on it (which changes the line endings)
Aaron Marcuse-Kubitza
11:47 AM Revision 7522: mappings/Makefile: VegCore.vocab.csv: Added back ambiguous terms, so that the vocabulary contains all terms defined by VegCore, regardless of whether they are ambiguous or unambiguous terms
Aaron Marcuse-Kubitza
11:44 AM Revision 7521: mappings/Makefile: VegCore.vocab.csv: Added back synonyms, so that the vocabulary contains all terms defined by VegCore, regardless of whether they are synonyms or primary terms. This also prevents VegCore.vocab.csv from losing entries when terms are renamed, which made it difficult to verify that no terms were lost when refactoring.
Aaron Marcuse-Kubitza
05:50 AM Revision 7520: inputs/MO/Specimen/postprocess.sql: Remove frameshifted rows by detecting InstitutionCodes without any letters
Aaron Marcuse-Kubitza
04:59 AM Revision 7519: inputs/ARIZ/Specimen/map.csv: CollectorNumber/FieldNumber: Use /_first to map these identical fields to the same location
Aaron Marcuse-Kubitza
04:54 AM Revision 7518: inputs/ARIZ/Specimen/map.csv: Fixed bug where the column names for InstitutionCode and CollectionCode were reversed in the source data
Aaron Marcuse-Kubitza
04:14 AM Revision 7517: inputs/*/Specimen/map.csv for Canadensys sources: Remapped institutionID to UNUSED
Aaron Marcuse-Kubitza

02/09/2013

07:45 AM Revision 7516: mappings/VegCore.htm: Regenerated from wiki. The original*, accepted*, and verbatim* Taxon fields have now been moved to separate OriginalTaxonDetermination, AcceptedTaxonDetermination, and TaxonVerbatim tables.
Aaron Marcuse-Kubitza
06:52 AM Revision 7515: mappings/VegCore.htm: Regenerated from wiki
Aaron Marcuse-Kubitza
06:34 AM Revision 7514: mappings/VegCore.htm: Regenerated from wiki
Aaron Marcuse-Kubitza
04:08 AM Revision 7513: README.TXT: Maintenance: VegCore data dictionary: Replaced VegCore.*.csv with VegCore.htm because now that VegCore.*.csv are sorted alphabetically, they generally don't change when VegCore.htm changes
Aaron Marcuse-Kubitza
04:04 AM Revision 7512: mappings/VegCore.*.csv: Regenerated from wiki. A plain text label is now used for Replace with, which fixes a bug where the PRIVATE permalink pointed to its Replace with in realLatitude instead of its definition.
Aaron Marcuse-Kubitza
03:55 AM Revision 7511: redmine_synonyms: Support plain text labels other than Alternative, such as Replace with
Aaron Marcuse-Kubitza
03:13 AM Revision 7510: mappings/VegCore.*.csv: Regenerated from wiki. Alternatives now contain the "Alternative" label as plain text rather than as an image title, thus avoiding an HTML anchor conflict with the definition and allowing ambiguous terms to be placed before their alternatives as well as after.
Aaron Marcuse-Kubitza
03:11 AM Revision 7509: README.TXT: Maintenance: VegCore data dictionary: Updated VegCore.csv filename to VegCore.*.csv
Aaron Marcuse-Kubitza
02:57 AM Revision 7508: redmine_synonyms: Support alternatives which contain the "Alternative" label as plain text rather than as an image title. This is done to include the "Alternative" label in the HTML anchor and thus prevent the anchor from conflicting with the actual definition of the alternative (which would otherwise have the same anchor text). This allows ambiguous terms to be placed before their alternatives as well as after, because there won't be anchor conflicts that need to be resolved with careful ordering.
Aaron Marcuse-Kubitza
02:48 AM Revision 7507: mappings/VegCore.csv: Regenerated from wiki. Taxon terms with prefixes for other TaxonDeterminations now indicate the analogous term in an "analogous to" label next to the term
Aaron Marcuse-Kubitza
02:47 AM Revision 7506: mappings/VegCore.csv: Regenerated from wiki. Taxon terms with prefixes for other TaxonDeterminations now indicate the analogous term in an "analogous to" label next to the term
Aaron Marcuse-Kubitza
02:47 AM Revision 7505: mappings/VegCore.csv: Regenerated from wiki. Taxon terms with prefixes for other TaxonDeterminations now indicate the analogous term in an "analogous to" label next to the term
Aaron Marcuse-Kubitza

02/07/2013

01:57 PM Revision 7504: mappings/VegCore-VegBIEN.csv: datasourceRecordID: Fixed bug where also need to add datasourceRecordID next to occurrenceID for an institutionCode remap switch
Aaron Marcuse-Kubitza
01:57 PM Revision 7503: inputs/bien_web/observation/test.xml.ref: Regenerated
Aaron Marcuse-Kubitza
01:48 PM Revision 7502: inputs/import.stats.xls: Updated import times using the import_times bugfix for times longer than a day
Aaron Marcuse-Kubitza
01:45 PM Revision 7501: import_times: times(): Fixed bug where need to match whitespace in times, in order to match times with days
Aaron Marcuse-Kubitza
12:00 PM Revision 7500: inputs/*/Specimen/map.csv: Remapped ID to datasourceRecordID
Aaron Marcuse-Kubitza
11:55 AM Revision 7499: mappings/VegCore-VegBIEN.csv: Mapped datasourceRecordID
Aaron Marcuse-Kubitza
11:51 AM Revision 7498: inputs/import.stats.xls: Updated import times
Aaron Marcuse-Kubitza
08:38 AM Revision 7497: inputs/FIA/_src/_README.TXT: Documented that the refresh is missing some PLT_CN values present in the original version
Aaron Marcuse-Kubitza
08:33 AM Revision 7496: inputs/FIA/import_order.txt: Reverted back to using FIA_COND_unique instead of COND_unique because the PLT_CN IDs in the refresh don't match the PLT_CN IDs in the original version, making COND_unique and Organism incompatible
Aaron Marcuse-Kubitza
08:27 AM Revision 7495: inputs/FIA/import_order.txt: Removed FIA_COND_unique, which is superseded by COND_unique
Aaron Marcuse-Kubitza
08:26 AM Revision 7494: inputs/FIA/import_order.txt: Fixed bug where need to import COND_unique before Organism because the plot entries need to be created before they can be linked to by organisms
Aaron Marcuse-Kubitza
07:25 AM Revision 7493: redmine_synonyms: sed pattern: Match <h# directly at the beginning of the line rather than after ^.*, which greatly speeds up the pattern matching because the first character is a literal character. (If <h# were not located at the left margin, the ^.* would unfortunately still be needed because the beginning of the line needs to be matched in order to be removed by the replacement operation.)
Aaron Marcuse-Kubitza
07:22 AM Revision 7492: mappings/VegCore.csv: Regenerated from wiki. Alternatives are now able to use h3 instead of h4 (which had display problems). realLatitude/Longitude is now no longer needs the ? prefix to have its replacement (PRIVATE) interpreted as an alternative, and thus is properly able to be included in the vocabulary.
Aaron Marcuse-Kubitza
07:16 AM Revision 7491: mappings/Makefile: VegCore.vocab.csv: Use the term's type label instead of its header level to determine if it's a synonym or alternative. This allows header levels to be chosen for presentational reasons rather than being constrained by being parsable.
Aaron Marcuse-Kubitza
07:05 AM Revision 7490: redmine_synonyms: Don't require ambiguous terms to start with ?, because the ambiguous term for an alternative can be identified simply by choosing the last term that didn't have a type label (previously, this would have been the last term that wasn't h3 or h4)
Aaron Marcuse-Kubitza
07:01 AM Revision 7489: redmine_synonyms: Use the term's type label instead of its header level to determine if it's a synonym or alternative. This allows header levels to be chosen for presentational reasons rather than being constrained by being parsable.
Aaron Marcuse-Kubitza
06:26 AM Revision 7488: mappings/VegCore.csv: Regenerated from wiki. The data dictionary has been reformatted to be much more vertically compact, by placing the term type (Synonym, Alternative, etc.) and sources (From:) on the same line as the term. Note that globalUniqueIdentifier_SpeciesLink has been removed from the vocabulary because a definition entry has been added for it (when this entry is missing, the term is incorrectly identified as a primary term).
Aaron Marcuse-Kubitza
06:21 AM Revision 7487: mappings/Makefile, redmine_synonyms: Updated for new VegCore data dictionary format, which prefixes the term type (Synonym, Alternative, etc.) to the term instead of including it as a section label. This ensures that the term type of a non-primary term is shown next to the term when it is visited via a permalink, which causes the term header to appear at the top of the screen and obscures the section header containing the type.
Aaron Marcuse-Kubitza
06:00 AM Revision 7486: mappings/Makefile: VegCore.thesaurus.csv: removal of tables: ignore errors if grep found no match
Aaron Marcuse-Kubitza
02:06 AM Revision 7485: Renamed mappings/VegCore.csv to VegCore.vocab.csv and Veg+-VegCore.csv to VegCore.thesaurus.csv for clarity
Aaron Marcuse-Kubitza
02:03 AM Revision 7484: mappings/Makefile, input.Makefile: Renamed $(dict) to $(thesaurus) because Veg+-VegCore.csv is actually a thesaurus, not a dictionary
Aaron Marcuse-Kubitza
01:57 AM Revision 7483: mappings/Makefile: Replaced occurrences of VegCore.csv with $(vocab) and Veg+-VegCore.csv with $(dict)
Aaron Marcuse-Kubitza

02/06/2013

07:34 PM Revision 7482: README.TXT: Maintenance: VegCore data dictionary: When moving terms, check that no terms were lost: Updated steps now that VegCore.csv and Veg+-VegCore.csv are sorted by name, so that a comparison of added/deleted counts is not necessary and a simple `svn di` can be used
Aaron Marcuse-Kubitza
07:33 PM Revision 7481: mappings/Makefile: Veg+-VegCore.csv: Sort terms by name so that reordering terms in the VegCore data dictionary does not cause Veg+-VegCore.csv to change. This makes it much easier to identify synonyms and ambiguous terms that were accidentally deleted during a data dictionary refactoring. (Note that these are no longer included in VegCore.csv, so this is required in addition to sorting VegCore.csv by name.)
Aaron Marcuse-Kubitza
07:26 PM Revision 7480: mappings/Makefile: VegCore.csv: Sort terms by name so that reordering terms in the VegCore data dictionary does not cause VegCore.csv to change. This makes it much easier to identify terms that were accidentally deleted during a data dictionary refactoring.
Aaron Marcuse-Kubitza

02/05/2013

06:19 PM Revision 7479: mappings/VegCore.csv: Regenerated from wiki. This adds cf_aff.
Aaron Marcuse-Kubitza
06:18 PM Revision 7478: mappings/Makefile: VegCore.csv: Filter out namespaces by matching only terms whose header links within the data dictionary
Aaron Marcuse-Kubitza
06:08 PM Revision 7477: mappings/VegCore.csv: Regenerated from wiki. This causes TNRS's Annotations (cf/aff) to be mapped into VegBIEN.
Aaron Marcuse-Kubitza
06:05 PM Revision 7476: mappings/VegCore-VegBIEN.csv: matched*Fit_fraction: Remapped to taxonconfidence instead of taxonfit
Aaron Marcuse-Kubitza
05:56 PM Revision 7475: mappings/Makefile: VegCore.csv: Fixed bug where need to remove duplicates, which are no longer supported by canon, by removing alternatives of ambiguous terms when these occur separately from their definitions
Aaron Marcuse-Kubitza
05:29 PM Revision 7474: mappings/Makefile: VegCore.csv: Removed synonyms and ambiguous terms, since the canonicalization of them is handled by Veg+-VegCore.csv. This also reduces the time it takes canon to build the in-memory Python dict of replacements, which scales to all inputs and should speed up the build/test command.
Aaron Marcuse-Kubitza
05:22 PM Revision 7473: mappings/Makefile: VegCore.csv: Removed synonyms, since the canonicalization of them is handled by Veg+-VegCore.csv
Aaron Marcuse-Kubitza
05:10 PM Revision 7472: mappings/Makefile: VegCore.csv: Match terms by header # instead of matching all anchors, in order to include the leading ? before an ambiguous term
Aaron Marcuse-Kubitza
04:42 PM Revision 7471: mappings/Makefile: Veg+-VegCore.csv: Generate dynamically from VegCore.htm, which allows the VegCore thesaurus to be automatically kept up to date. More importantly, it allows terms in all map spreadsheets to be updated simultaneously when a term is renamed (e.g. by replacing a term with one of its synonyms).
Aaron Marcuse-Kubitza
04:40 PM Revision 7470: mappings/VegX-VegCore.csv: Applied term renamings from the new dynamically generated Veg+-VegCore.csv. Updates to VegCore term names that have occurred since the data dictionary was created are now able to take effect, which involves remapping several fields.
Aaron Marcuse-Kubitza
04:32 PM Revision 7469: mappings/VegCore-VegBIEN.csv, inputs/*/*/map.csv: Applied term renamings from the new dynamically generated Veg+-VegCore.csv, which reflects the current state of the data dictionary. (Permanently switching to the new Veg+-VegCore.csv will be a separate change.) Updates to VegCore term names that have occurred since the data dictionary was created are now able to take effect, which involves remapping and inferring units on several fields.
Aaron Marcuse-Kubitza
04:27 PM Revision 7468: mappings/VegCore-VegBIEN.csv: Mapped basalDiameter_in
Aaron Marcuse-Kubitza
04:15 PM Revision 7467: mappings/VegCore-VegBIEN.csv: Mapped diameterBreastHeightGentry_cm, basalDiameter_cm, precipitation_mm
Aaron Marcuse-Kubitza
04:14 PM Revision 7466: schemas/vegbien.sql: Added _mm_to_m()
Aaron Marcuse-Kubitza
03:56 PM Revision 7465: mappings/Makefile: Veg+-VegCore.csv: Fixed bugs where also need to filter out ambiguous tables, but shouldn't filter out acronyms (which are regular fields)
Aaron Marcuse-Kubitza
03:40 PM Revision 7464: mappings/VegCore-VegBIEN.csv: locationID->location.sourceaccessioncode: Removed restriction that this mapping can't occur if geovalidation information is present. The locationID is no longer mapped to the place.sourceaccessioncode, so this filter is not necessary.
Aaron Marcuse-Kubitza
03:38 PM Revision 7463: mappings/VegCore.csv: Regenerated from wiki
Aaron Marcuse-Kubitza
03:19 PM Revision 7462: mappings/Makefile: Veg+-VegCore.csv: Fixed bug where need to filter out table names to avoid applying table replacements to fields which have the same name as a table
Aaron Marcuse-Kubitza
03:03 PM Revision 7461: inputs/Madidi/map.csv: Fixed bug where needed to remove duplicate input names, now that translate doesn't allow them
Aaron Marcuse-Kubitza
01:59 PM Revision 7460: mappings/Makefile: VegX-VegCore.csv: Sort by the input column instead of the output column to keep the sort order stable across VegCore term renames
Aaron Marcuse-Kubitza
01:46 PM Revision 7459: mappings/Makefile: Veg+-VegCore.csv: Before running collapse_multimap, canonicalize alternatives of ambiguous terms using unambiguous mappings. This ensures that the alternatives lists contain only canonical VegCore terms rather than synonyms.
Aaron Marcuse-Kubitza
01:43 PM Revision 7458: mappings/VegCore.csv: Regenerated from wiki. All synonyms are now hyperlinked, allowing them to be matched by redmine_synonyms.
Aaron Marcuse-Kubitza
01:31 PM Revision 7457: mappings/Veg+-VegCore.csv: Removed Sources, Definition columns because source information is now in the VegCore data dictionary
Aaron Marcuse-Kubitza
01:25 PM Revision 7456: mappings/VegCore.csv: Regenerated from wiki. Ambiguous terms newly available to redmine_synonyms due to the bugfix now have multiple alternatives.
Aaron Marcuse-Kubitza
01:25 PM Revision 7455: redmine_synonyms: Ambiguous terms: Fixed bug where need to use header # instead of term name to determine whether a term is an alternative, because some alternatives (e.g. verbatimElevation) don't follow the units-suffix naming convention.
Aaron Marcuse-Kubitza
12:58 PM Revision 7454: mappings/VegCore.csv: Regenerated from wiki. All ambiguous terms now have multiple alternatives, preventing them from being automapped to a single alternative without prompting the user for confirmation
Aaron Marcuse-Kubitza
12:50 PM Revision 7453: mappings/Makefile: Veg+-VegCore.csv: translate: Fixed bug where need to run on $@ instead of $<
Aaron Marcuse-Kubitza
12:49 PM Revision 7452: mappings/VegCore.csv: Regenerated from wiki. All ambiguous terms now have multiple alternatives, preventing them from being automapped to a single alternative without prompting the user for confirmation
Aaron Marcuse-Kubitza
12:22 PM Revision 7451: mappings/VegCore.csv: Regenerated from wiki. All mappings/Veg+-VegCore.csv terms are now added as synonyms or separate terms.
Aaron Marcuse-Kubitza
10:26 AM Revision 7450: mappings/VegCore.csv: Regenerated from wiki. Most ambiguous terms are now split into alternatives, and most mappings/Veg+-VegCore.csv terms are now added as synonyms.
Aaron Marcuse-Kubitza
06:12 AM Revision 7449: canon: Raise an error if two input terms map to the same simplified string
Aaron Marcuse-Kubitza
04:34 AM Revision 7448: translate: Changed dictionary to thesaurus, since the map used actually has synonyms rather than definitions
Aaron Marcuse-Kubitza
04:31 AM Revision 7447: mappings/Makefile: Veg+-VegCore.csv: Translate the thesaurus's output terms using itself in order to map a synonym of an ambiguous term directly to its alternatives list rather than only to the ambiguous term itself
Aaron Marcuse-Kubitza
04:26 AM Revision 7446: mappings/Makefile: Veg+-VegCore.csv: Run collapse_multimap on the generated map so that all alternatives are included, rather than just the first alternative, when translate maps an ambiguous term
Aaron Marcuse-Kubitza
04:25 AM Revision 7445: redmine_synonyms: Fixed bug where need to output a CSV rather than TSV to be usable by other programs that use map spreadsheets
Aaron Marcuse-Kubitza
04:23 AM Revision 7444: Added collapse_multimap, which collapses multimap entries in a spreadsheet dictionary
Aaron Marcuse-Kubitza
03:45 AM Revision 7443: mappings/Veg+-VegCore.csv: Separate alternatives of ambiguous terms with , instead of ", " for easier machine-parsability
Aaron Marcuse-Kubitza
03:31 AM Revision 7442: redmine_synonyms: Added support for ambiguous terms, which unlike the synonyms format nests the term (the alternative) under the synonym (the ambiguous term) rather than the synonym under the term. Note that ambiguous terms must also be prefixed with ? to differentiate them from composites (e.g. recordedBy_givenName), which use the same _-based naming convention.
Aaron Marcuse-Kubitza
03:08 AM Revision 7441: mappings/VegCore.csv: Regenerated from wiki
Aaron Marcuse-Kubitza
02:49 AM Revision 7440: mappings/VegCore.csv: Regenerated from wiki
Aaron Marcuse-Kubitza
02:22 AM Revision 7439: schemas/vegbien.sql: analytical_stem_view: Renamed scientificNameWithMorphospecies to taxonNameWithMorphospecies because it does not contain the scientific name author, as required by DwC scientificName <http://rs.tdwg.org/dwc/terms/#scientificName>
Aaron Marcuse-Kubitza
01:56 AM Revision 7438: mappings/Makefile: VegCore.tables.csv: Exclude ambiguous table names, which should not be part of the tables summary (as neither are table synonyms)
Aaron Marcuse-Kubitza
01:51 AM Revision 7437: input.Makefile: $(translate?): Merged with $(translate), which is not used independently
Aaron Marcuse-Kubitza
01:50 AM Revision 7436: input.Makefile: Use new translate_ci instead of translate
Aaron Marcuse-Kubitza
01:47 AM Revision 7435: mappings/Makefile: Use new translate_ci instead of translate
Aaron Marcuse-Kubitza
01:39 AM Revision 7434: Added translate_ci
Aaron Marcuse-Kubitza

02/04/2013

11:03 PM Revision 7433: mappings/VegCore-VegBIEN.csv: institutionCode list->sourcename mapping: _split(): Also match ; as a separator, and match separators with or without a following space
Aaron Marcuse-Kubitza

02/02/2013

05:39 PM Revision 7432: mappings/Makefile: Added target to create Veg+-VegCore.csv from VegCore.htm, initially commented out until all the synonyms in the existing Veg+-VegCore.csv are added to the VegCore data dictionary <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore_data_dictionary>
Aaron Marcuse-Kubitza
05:38 PM Revision 7431: Added redmine_synonyms, which translates a Redmine HTML page to a thesaurus
Aaron Marcuse-Kubitza
04:37 PM Revision 7430: lockfile: Linux: Documented why newgrp and recursive invocation of lockfile are needed
Aaron Marcuse-Kubitza
04:33 PM Revision 7429: lockfile: Linux: Fixed bug where need to change primary group of the dotlockfile process to the group of the dir to contain the lockfile, because dotlockfile otherwise reports a "permission denied" error (even though the directory is actually writable, dotlockfile thinks it isn't). Running dotlockfile with a different primary group is complicated because newgrp, the command that does this, does not pass arguments to the new process, so they must instead be passed via environment variables and a recursive invocation of lockfile (with the $inner recursion flag set). Additionally, exec cannot be used to propagate the PPID (needed by dotlockfile) because newgrp creates a new process rather than using exec, so it must be manually entered into the lockfile after dotlockfile runs.
Aaron Marcuse-Kubitza
02:41 PM Revision 7428: lockfile: Linux: Fixed bug where need to lower retry count to avoid overflowing the retries variable
Aaron Marcuse-Kubitza
02:37 PM Revision 7427: lockfile: Linux: Added workaround for bug in dotlockfile where using -1 to retry indefinitely doesn't work, so need to use large integer instead
Aaron Marcuse-Kubitza
01:49 PM Revision 7426: lockfile: Linux: Use bin/dotlockfile instead of the system's dotlockfile, because the system's dotlockfile is SETGID mail, which prevents it from creating lockfiles in a directory owned by the bien user and group when being run by the login user
Aaron Marcuse-Kubitza
01:38 PM Revision 7425: bin/: svn:ignore: Added dotlockfile, which is copied from the system during installation
Aaron Marcuse-Kubitza
01:30 PM Revision 7424: bin/: svn:ignore: Removed no longer applicable test_output
Aaron Marcuse-Kubitza
01:26 PM Revision 7423: root Makefile: misc-Linux: Added command to copy dotlockfile to the bin/ dir, so that it can be used without being SETGID mail, which would prevent it from creating lockfiles in a directory owned by the bien user and group when being run by the user
Aaron Marcuse-Kubitza
01:24 PM Revision 7422: root Makefile: core: Added misc-* to install other dependencies
Aaron Marcuse-Kubitza
11:56 AM Revision 7421: schemas/vegbien.sql: analytical_stem_view: scientificNameWithMorphospecies: Removed no longer needed canon_taxonverbatim.family alternative, since the family will be included in the canon_taxonlabel.taxonomicname by the mappings
Aaron Marcuse-Kubitza
11:49 AM Revision 7420: schemas/vegbien.sql: analytical_stem_view: scientificNameWithMorphospecies: Fixed bug where need to use canon_*taxonlabel*.taxonomicname instead of canon_taxonverbatim.taxonomicname as one of the alternatives because only canon_taxonlabel.taxonomicname is guaranteed to be populated by the mappings, while canon_taxonverbatim.taxonomicname will only be populated if the datasource explicitly specifies that field. This distinction is only meaningful for data without a TNRS match, as TNRS supplies canon_taxonverbatim.taxonomicname.
Aaron Marcuse-Kubitza
11:28 AM Revision 7419: import_all: after_import(): Added wait on tnrs.make's lockfile to ensure that all background scrubbing processes are complete before creating the analytical DB
Aaron Marcuse-Kubitza
11:18 AM Revision 7418: import_all: Moved `waitpid $jobs` into after_import()
Aaron Marcuse-Kubitza

02/01/2013

04:57 PM Revision 7417: schemas/vegbien.ERD.mwb: Fixed table sizes
Aaron Marcuse-Kubitza
04:51 PM Revision 7416: schemas/vegbien.ERD.mwb: Regenerated exports
Aaron Marcuse-Kubitza
04:34 PM Revision 7415: schemas/vegbien.sql: removed all accessioncode fields, as VegBIEN does not use them
Aaron Marcuse-Kubitza
03:10 PM Revision 7414: Added inputs/FIA/_src/FIADB_version4.accdb and FIADB_version4.sql (created from it using Access To PostgreSQL and the additional transformations at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Tools#MS-Access-database-MDB>)
Aaron Marcuse-Kubitza

01/31/2013

08:20 PM Revision 7413: Added inputs/FIA/COND_unique/, generated from new FIA data
Aaron Marcuse-Kubitza
08:05 PM Revision 7412: inputs/FIA/FIA_COND_unique/create.sql: Fixed bug where need to remove `CREATE TABLE :table AS` at beginning because that is added by the make target
Aaron Marcuse-Kubitza
08:03 PM Revision 7411: inputs/FIA/geoscrub.~.clean_up.sql: Moved creation of FIA_COND_unique to FIA_COND_unique/create.sql
Aaron Marcuse-Kubitza
07:40 PM Revision 7410: README.TXT: Full database import: Updated time until import_all returns control to the shell to account for the TNRS names now being imported concurrently with the inputs rather than before them
Aaron Marcuse-Kubitza
07:31 PM Revision 7409: mappings/VegCore-VegBIEN.csv: Also include morphospecies in the accepted taxondetermination's taxonverbatim, so that it can easily be retrieved by the analytical DB views
Aaron Marcuse-Kubitza
07:15 PM Revision 7408: schemas/vegbien.sql: analytical_stem_view: scientificNameWithMorphospecies: Fixed bug where need to use the taxonName or scientificName when the name components are not provided, as is the case when there is no scrubbed taxondetermination (because TNRS returns no match)
Aaron Marcuse-Kubitza
06:08 PM Revision 7407: mappings/VegCore.csv: Regenerated from wiki. This adds Brad's DwC ID terms and their definitions in <https://projects.nceas.ucsb.edu/nceas/attachments/download/621/vegbien_identifier_examples.xlsx>.
Aaron Marcuse-Kubitza
05:06 PM Revision 7406: schemas/vegbien.ERD.mwb: Regenerated exports
Aaron Marcuse-Kubitza
04:04 PM Revision 7405: join: Added support for direct mappings to VegBIEN by passing through outputs that start with / (indicating an XPath rather than a term)
Aaron Marcuse-Kubitza
04:01 PM Revision 7404: mappings/VegCore.csv: Regenerated from wiki
Aaron Marcuse-Kubitza
11:38 AM Revision 7403: schemas/vegbien.sql: analytical_stem_view: Added family_matched, taxonName_matched, scientificNameAuthorship_matched
Aaron Marcuse-Kubitza
11:02 AM Revision 7402: schemas/vegbien.sql: analytical_stem_view: Added family_verbatim, scientificName_verbatim, scientificNameAuthorship_verbatim from datasource taxondetermination
Aaron Marcuse-Kubitza
10:57 AM Revision 7401: mappings/VegCore.csv: Regenerated from wiki
Aaron Marcuse-Kubitza
10:30 AM Revision 7400: schemas/vegbien.sql: analytical_stem_view: Fixed bug where need to use identifiedBy and dateIdentified from the *datasource* taxondetermination rather than the canonical taxondetermination (whichever taxondetermination is most scrubbed)
Aaron Marcuse-Kubitza
10:23 AM Revision 7399: schemas/vegbien.sql: taxondetermination: taxondetermination_set_iscurrent(): is_datasource_current: Fixed bug where need to filter out determinationtypes for matched/accepted determinations, which are not datasource determinations
Aaron Marcuse-Kubitza
10:19 AM Revision 7398: schemas/vegbien.sql: taxondetermination: taxondetermination_set_iscurrent(): Fixed bug where need to also set existing datasource_current taxondetermination's is_datasource_current to false
Aaron Marcuse-Kubitza
08:52 AM Revision 7397: xml_dom.py: replace_with_text(): Added support for all scalar (non-Node) types, which will be stringified using strings.ustr()
Aaron Marcuse-Kubitza
03:52 AM Revision 7396: schemas/functions.sql: Added _fix_date()
Aaron Marcuse-Kubitza
02:49 AM Revision 7395: sql_io.py: put_table(): Documented that much of the complexity of the normalizing algorithm is due to PostgreSQL not having a native command for insert/on duplicate select
Aaron Marcuse-Kubitza
02:24 AM Revision 7394: sql_io.py: put_table(): Corrected "insert/if not exists get" to "insert/on duplicate select"
Aaron Marcuse-Kubitza
01:52 AM Revision 7393: sql_io.py: put_table(): Removed no longer applicable requirement that it be run at the beginning of a transaction, which was only required when the output table was locked during the function call
Aaron Marcuse-Kubitza
01:48 AM Revision 7392: sql_io.py: put_table(): Documented that the function's insert/if not exists get algorithm does not support database triggers that populate fields covered by a unique constraint
Aaron Marcuse-Kubitza
01:42 AM Revision 7391: inputs/FIA/_src/_README.TXT: Documented that FIA does not provide data for some states, e.g. HI
Aaron Marcuse-Kubitza

01/30/2013

10:48 PM Revision 7390: config/: Set svn:ignore to exclude *password files
Aaron Marcuse-Kubitza
10:41 PM Revision 7389: Removing config/bien_read_password from version control
Aaron Marcuse-Kubitza
10:30 PM Revision 7388: Removing config/bien_password from version control
Aaron Marcuse-Kubitza

01/29/2013

03:26 PM Revision 7387: inputs/FIA/: Added refreshed data (not yet mapped)
Aaron Marcuse-Kubitza
03:15 PM Revision 7386: input.Makefile: Existing maps discovery: $(exts): Also match uppercase versions of extensions
Aaron Marcuse-Kubitza
03:12 PM Revision 7385: lib/common.Makefile: Added $(ucase) and $(ci)
Aaron Marcuse-Kubitza
01:56 PM Revision 7384: inputs/FIA/_src/Makefile: Table bundling: $(tableCsvs): Fixed bug where need to replace % with $* in $(csvPattern)
Aaron Marcuse-Kubitza
01:15 PM Revision 7383: inputs/FIA/_src/Makefile: Table bundling: Fixed bug where need to remove trailing slashes from dirs that will match a target pattern
Aaron Marcuse-Kubitza
01:09 PM Revision 7382: inputs/FIA/_src/Makefile: Added Table bundling targets to regroup CSVs by tables
Aaron Marcuse-Kubitza
01:09 PM Revision 7381: lib/common.Makefile: Added $(mkdir)
Aaron Marcuse-Kubitza
11:02 AM Revision 7380: Added inputs/FIA/_src/_README.TXT with Bob's comments
Aaron Marcuse-Kubitza
11:02 AM Revision 7379: input.Makefile: SVN: $(_svnFilesGlob): Added README.TXT
Aaron Marcuse-Kubitza
10:33 AM Revision 7378: mappings/VegCore.csv: Regenerated from wiki. Synonym lists have now been translated to sections to create a web page anchor for each synonym, using the steps at <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/VegCore_refactoring#Index-synonyms-as-web-page-anchors>. This enables searching for VegCore synonyms in the data dictionary as well as terms, and makes it possible to swap a term and a synonym while still keeping both as indexed anchors.
Aaron Marcuse-Kubitza
06:19 AM Revision 7377: mappings/VegCore.csv: Regenerated from wiki. All uncategorized terms have now been moved to tables.
Aaron Marcuse-Kubitza
06:19 AM Revision 7376: README.TXT: Maintenance: VegCore data dictionary: Added steps to check that no terms were lost when moving terms
Aaron Marcuse-Kubitza

01/28/2013

05:13 PM Revision 7375: inputs/import.stats.xls: Updated import times
Aaron Marcuse-Kubitza
05:12 PM Revision 7374: mappings/VegCore.csv: Regenerated from wiki
Aaron Marcuse-Kubitza

01/25/2013

03:54 PM Revision 7373: schemas/vegbien.sql: analytical_stem_view: coordinates: Only use county_centroids coordinates when datasource coordinates are not provided, not also when datasource coordinates aren't geovalid. This also fixes a bug where (NULL) county_centroids coordinates were used for non-geovalid coordinates even when there was no county_centroids match, rather than including the non-geovalid coordinates.
Aaron Marcuse-Kubitza
03:34 PM Revision 7372: mappings/VegCore.csv: Regenerated from wiki
Aaron Marcuse-Kubitza
11:27 AM Revision 7371: schemas/vegbien.sql: taxondetermination: Added is_datasource_current, which is autopopulated to the most recent *datasource-provided* taxondetermination
Aaron Marcuse-Kubitza
11:07 AM Revision 7370: schemas/vegbien.sql: taxondetermination: Added taxondetermination_single_accepted_determination unique index to facilitate joining on the accepted determination
Aaron Marcuse-Kubitza
11:05 AM Revision 7369: schemas/vegbien.sql: taxondetermination: Added taxondetermination_single_matched_determination unique index to facilitate joining on the matched determination
Aaron Marcuse-Kubitza
10:32 AM Revision 7368: schemas/vegbien.sql: taxondetermination: Removed notespublic, notesmgt, which are not used by VegBIEN
Aaron Marcuse-Kubitza
09:30 AM Revision 7367: schemas/vegbien.sql: taxon_trait_view: scientificName: Use taxonverbatim.taxonname when taxonlabel/taxonverbatim.taxonomicname are not provided, to accommodate TNRS names. This is part of the workaround for the bug where the taxonlabel's taxonomicname (concatenated taxonomicname) is occasionally not populated.
Aaron Marcuse-Kubitza
09:10 AM Revision 7366: schemas/vegbien.sql: taxon_trait_view: Added workaround for bug where the taxonlabel's taxonomicname (concatenated taxonomicname) is occasionally not populated due to a taxonlabel constraint violation, by using the taxonverbatim's taxonomicname instead in these cases. This bug, which appeared in the r7317 import, is so far not reproducible (tested on Mac OS X), so its cause is unknown, but may be caused by a bug in functions._merge_prefix(), which is run on the taxonlabel's taxonomicname but not the taxonverbatim's taxonomicname.
Aaron Marcuse-Kubitza

01/24/2013

09:51 PM Revision 7365: schemas/vegbien.sql: analytical_stem_view: Added dateIdentified, identificationRemarks per Brad's request (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#E-mail-on-2013-1-16)
Aaron Marcuse-Kubitza
09:40 PM Revision 7364: inputs/FIA/_src/Makefile: Added extraction targets to extract zip archives
Aaron Marcuse-Kubitza
09:07 PM Revision 7363: inputs/FIA/_src/download: Use new Makefile, which uses make logic to determine if a file needs to be downloaded
Aaron Marcuse-Kubitza
09:05 PM Revision 7362: Added inputs/FIA/_src/Makefile, with targets to download each zip archive
Aaron Marcuse-Kubitza
08:00 PM Revision 7361: schemas/vegbien.sql: analytical_stem_view: derived terms: Added _bien suffix per Brad's request (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Spot-checking#Brad-Boyles-comments)
Aaron Marcuse-Kubitza
03:22 PM Revision 7360: Added inputs/FIA/_src/FIADB_version4.accdb.url
Aaron Marcuse-Kubitza
03:18 PM Revision 7359: inputs/FIA/_src/download: Only run wget on files that don't yet exist
Aaron Marcuse-Kubitza
03:16 PM Revision 7358: inputs/FIA/_src/download: Run wget in same directory as script to ensure files get downloaded there
Aaron Marcuse-Kubitza
03:06 PM Revision 7357: inputs/FIA/_src/download: Set svn:executable
Aaron Marcuse-Kubitza
03:04 PM Revision 7356: Added inputs/FIA/_src/download to download archives of CSVs for each state
Aaron Marcuse-Kubitza
03:03 PM Revision 7355: to_do/timeline.2013.xls: Updated with changes during conference call
Aaron Marcuse-Kubitza
09:46 AM Revision 7354: schemas/vegbien.sql: taxon_trait_view: Renamed datasource_taxonverbatim to taxonverbatim because there is now only one taxonverbatim
Aaron Marcuse-Kubitza
09:31 AM Revision 7353: schemas/vegbien.sql: taxon_trait_view: Moved the taxondetermination.iscurrent filter to the join condition to allow using the taxondetermination_single_current_determination index
Aaron Marcuse-Kubitza
09:24 AM Revision 7352: schemas/vegbien.sql: taxon_trait_view: Join only on the primary taxonlabel, not the accepted taxonlabel, because the scrubbed name is now available directly via the taxonlabel attached to the scrubbed taxondetermination
Aaron Marcuse-Kubitza
09:11 AM Revision 7351: schemas/vegbien.sql: analytical_stem_view: Added locality
Aaron Marcuse-Kubitza
08:18 AM Revision 7350: inputs/UNCC/Specimen/map.csv: accession: Remapped to catalogNumber per Bob's corrections
Aaron Marcuse-Kubitza

01/23/2013

10:31 PM Revision 7349: schemas/vegbien.ERD.mwb: Regenerated exports
Aaron Marcuse-Kubitza
10:25 PM Revision 7348: mappings/VegCore.csv: Regenerated from wiki
Aaron Marcuse-Kubitza
10:01 PM Revision 7347: README.TXT: Schema changes: Added instructions to run the appropriate sync function when changing the analytical views
Aaron Marcuse-Kubitza
09:56 PM Revision 7346: schemas/vegbien.sql: analytical_stem_view: Added georeferenceProtocol, which is set to 'county centroid' when county centroid coordinates are used
Aaron Marcuse-Kubitza
08:12 PM Revision 7345: make_analytical_db: Don't run export_analytical_db if the SQL script exits with an error
Aaron Marcuse-Kubitza
08:04 PM Revision 7344: README.TXT: Full database import: record the import times in inputs/import.stats.xls: Added `export version=<version>` because import_times may be run in a shell different from the one that the import was run in
Aaron Marcuse-Kubitza
08:03 PM Revision 7343: inputs/import.stats.xls: Updated import times
Aaron Marcuse-Kubitza
 

Also available in: Atom