web/links/index.htm: updated to Firefox bookmarks. git: added instructions to migrate from svn.
web/links/index.htm: updated to Firefox bookmarks. added instructions for running `svn switch` after moving everything into /trunk/ (for us, use `svn switch --force https://code.nceas.ucsb.edu/code/projects/bien/trunk # runtime: <1 min`).
/: re-set svn:ignore on new root (/trunk/)
moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).
added standard svn dirs branches, tags, trunk
web/links/index.htm: updated to Firefox bookmarks. added links to the sPlot ERD. Dropbox: added information about a missing upgrade notification when you use the Dropbox client to look up a file on the Dropbox website.
/README.TXT: added note that shell scripts should always be read-only, so that editing them while an import is in progress will not crash the import (see http://vegpath.org/links/#**%20modifying%20a%20running%20shell%20script)
/: svn:ignore the new extern/ dir, which will contain non-UCSB files
bugfix: inputs/.TNRS/schema.sql: scrubbed_family: Name_matched_accepted_family was missing from the TNRS results at one point, so we are now using Family_matched as a workaround to populate this. the workaround is for accepted names only, as no opinion names do not have an Accepted_name_family to prepend to the scrubbed name to parse.
inputs/.TNRS/schema.sql: reexported from live DB, which changes the element order
schemas/vegbien.sql: plot.**, analytical_stem_view, analytical_plot: added project_contributors
schemas/vegbien.sql: added project_contributors()
inputs/VegBank/import_order.txt: added projectcontributor_
inputs/VegBank/projectcontributor_/map.csv, postprocess.sql: added project_participant
mappings/VegCore-VegBIEN.csv: mapped project_participant
mappings/VegCore.htm: regenerated from wiki. merged confusingly-named TaxonObservation into TaxonOccurrence. added project_participant.
added inputs/VegBank/projectcontributor_/
inputs/VegBank/vegbank.~.clean_up.sql: projectcontributor.surname: prepend table name to avoid join collisions
inputs/VegBank/vegbank.~.clean_up.sql, inputs/CVS/cvs.~.clean_up.sql: Prevent "column name specified more than once" errors when tables are joined: put tables in alphabetical order for consistency
web/links/index.htm: updated to Firefox bookmarks. Quick links: removed page descriptions, which clutter up the quick links list.
web/links/index.htm: updated to Firefox bookmarks. Quick links: added VegBIEN FAQ.
bugfix: bin/boldify: also match [[]]-style links at the beginning and end of a line
bin/boldify: made it idempotent
bugfix: bin/boldify: fixed extended regular expression syntax, which doesn't support a \] inside [] (you instead have to put the ] right after the opening [^ )
added bin/boldify, which makes Redmine links bold
web/links/index.htm: updated to Firefox bookmarks. Quick links: removed page descriptions, which clutter up the quick links list
web/links/index.htm: updated to Firefox bookmarks. Quick links: added link to meetings
web/links/index.htm: updated to Firefox bookmarks. Quick links: reordered in the order they should apper to visitors to the BIEN links page, rather than to me in the bookmarks toolbar.
web/links/index.htm: updated to Firefox bookmarks. updated favicons.
web/links/index.htm: updated to Firefox bookmarks. added links for sPlot and iPlant open-sourcing.
inputs/publishable datasources.xlsx: updated
inputs/datasource_release_status.xlsx: renamed to `publishable datasources.xlsx` to match the spreadsheet title
planning/meetings/BIEN conference call availability.xlsx: updated
/README.TXT: to synchronize a Mac's settings with my testing machine's: added step to remove the downloaded Spam folder, because spam e-mails often contain viruses that would trigger clamscan
fix: planning/timeline/timeline.xls: realigned legend
planning/timeline/timeline.xls: hid previous weeks
planning/timeline/timeline.xls: crossed out and hid completed tasks
planning/timeline/timeline.xls: updated for progress
web/links/index.htm: updated to Firefox bookmarks. open-sourcing BIEN: added links for VegBank's license (GPLv2), and how these terms apply to us (a diff of our changes is not GPL-ed under GPLv2, although it is claimed to be GPL-ed under GPLv3)
inputs/VegBank/^taxon_observation.**.sample/create.sql, map.csv: added new project columns
inputs/VegBank/taxon_observation.**/postprocess.sql: added the project table
mapped inputs/VegBank/project/, which includes the projectName for attribution
inputs/CVS/^taxon_observation.**.sample/create.sql, map.csv: added new project columns
inputs/CVS/taxon_observation.**/postprocess.sql: added the project table
inputs/CVS/project/map.csv: mapped stopDate->projectEndDate
mapped inputs/CVS/project/, which includes the projectName for attribution
inputs/VegBIEN/Redmine/svn/.htaccess: updated to use much faster direct repository URL rather than Redmine web interface, now that the repository itself is publicly accessible in addition to the Redmine view of it
planning/timeline/timeline.2013.xls: renamed to timeline.xls so that this can continue to be used for 2014 (leaving a symlink from the old filename to preserve permalinks)
fix: inputs/TEX/Specimen*/map.csv, postprocess.sql: habitat: also placed in occurrenceRemarks so that this field gets parsed for growth form information, as requested by Brad (wiki.vegpath.org/TEX_validation#2013-2-26)
fix: inputs/TEX/Specimen*/map.csv: mapped constant values for specimenHolderInstitutions, country. these have to be added with `rm=1 ./inputs/TEX/Specimen.../run postprocess`.
bugfix: inputs/TEX/Specimen2/map.csv: mapped BARCODE to accessionNumber so that we have a unique ID for each row
schemas/vegbien.sql: analytical_stem_view: scientificName_verbatim: don't use taxonverbatim.taxonname+author as the scientificName_verbatim if only the author is provided. (this lead to weird scientificName_verbatims that contain just the author.)
inputs/datasource_release_status.xlsx: updated
web/links/index.htm: updated to Firefox bookmarks. added links for fixing the "App Store is temporarily unavailable" error (turn on Spotlight) and modifying a running shell script (unlink it first).
bugfix: bin/map: in_is_db: don't ignore errors when the table does not exist, because these prevent an errexit and allow an import to continue when a staging table is missing. suppressing this error had previously been necessary because metadata-only tables (Source/) used to not have installed staging tables, and the program had to react accordingly.
inputs/CVS/^taxon_observation.**.sample/create.sql: added Mike Lee's additional plots used to validate confidentiality-related fields (wiki.vegpath.org/CVS_validation#plots-to-include)
bugfix: inputs/CVS/^taxon_observation.**.sample/create.sql: include taxonName in the subset of columns that's imported for the validation, because it is _alt-ed with scientificName for forming the TNRS input name. this is unique to CVS, which is why it was not part of the validation subset copied from the VegBank subset.
/README.TXT: Full database import: documented that you should always start with a clean shell, which does not have changes to the env vars. (there have been inexplicable bugs that went away after closing and reopening the terminal window.) note that running `exec bash` is not sufficient to reset the env vars.
fix: lib/sh/util.sh: verbosity_min(): usage: clarified that '' is a special value that causes $verbosity to be overwritten to ''
lib/runscripts/table.run: added test_() target and use it in remake_VegBIEN_mappings() (it would not be clear that remake_VegBIEN_mappings() runs the tests)
bugfix: inputs/.TNRS/schema.sql: granted bien_read SELECT access to derived views as well as the core tnrs table
updated inputs/datasource_release_status.xlsx
added inputs/datasource_release_status.xlsx, export of Google spreadsheet at https://docs.google.com/spreadsheet/ccc?key=0ArZXrTAXd-TYdDRRb2RxYi11TWZrQVh5bVdKOURCeFE
planning/timeline/timeline.2013.xls: updated for progress
bugfix: schemas/vegbien.sql: location: use the place_id from the parent location when no place_id is specified. this fixes a bug in analytical_stem_view where the parent location's place_id was used because it was sometimes missing from the sublocation, but the parent place_id itself was sometimes missing instead if sublocations each had their own place information. this way, it is always available directly in the sublocation, populated from the parent location if needed.
bugfix: schemas/vegbien.sql: location: added place_id which is autopopulated from the current locationplace. join on this in plot.**, to avoid a 1:many join when a location has multiple locationplaces.
bugfix: schemas/vegbien.sql: locationevent_unique_within_parent_by_location unique index: need COALESCE around location_id since it's nullable
fix: inputs/CVS/^taxon_observation.**.sample/: added _no_import because this table duplicates part of what's imported from taxon_observation.**
bugfix: inputs/VegBank/plot/: added _no_import because this table is left-joined and should not be imported separately
bugfix: inputs/{.NCBI,CTFS}/*.src/: added _no_import because these tables are left-joined and should not be imported separately
inputs/import.stats.xls: removed table names from datasources where only one table is imported
fix: inputs/import.stats.xls: removed deleted tables from current import
inputs/import.stats.xls: updated import times
updated backups/TNRS.backup.md5
added backups/vegbien.r11786.backup.md5
/README.TXT: Full database import: backups: added step to download backup to local machine
bugfix: /Makefile: install: need to run inputs/download in live mode so that the flat files are actually downloaded
lib/common.Makefile: added %/live, for use with `make inputs/download`
planning/timeline/timeline.2013.xls: rescheduled tasks
/README.TXT: Full database import: In PostgreSQL: documented that the tables to check are located in the r# schema, not public
planning/timeline/timeline.2013.xls: datasource validations: reordered datasources according to Brian Enquist's new validation order (wiki.vegpath.org/Spot-checking_validation_order)
fix: schemas/vegbien.sql: analytical_specimen: added specimens-related columns that are in analytical_plot
inputs/GBIF/raw_occurrence_record_plants/map.csv: row_num: remapped to plain *row_num, like the other datasources that have this field
inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: rerun time: noted that this is only fast after manual vacuuming of the table (to remove the deleted rows from the index). autovacuum apparently does not run, although it should.
planning/timeline/timeline.2013.xls: hid previous weeks
planning/timeline/timeline.2013.xls: added timespan dots ◦ for supertasks
planning/timeline/timeline.2013.xls: legend: changed to movable text box to avoid needing to erase and repopulate the header columns with the legend cells
planning/timeline/timeline.2013.xls: crossed out and hid completed tasks
inputs/GBIF/raw_occurrence_record_plants/test.xml.ref: reran test, which added yearCollected/monthCollected/dayCollected
inputs/CVS/plantConcept_/create.sql: documented runtime (3 min)
inputs/CTFS/*.src/: added test.xml.ref
inputs/CTFS/*.src/: added VegBIEN.csv
bugfix: inputs/CTFS/TaxonOccurrence*/map.csv: things mapped to taxonObservationID: remapped to taxonOccurrenceID since taxonObservationID is not mapped to anything in VegBIEN (denormalized VegCore doesn't distinguish between taxon occurrences and taxon observations of them)
bugfix: inputs/ARIZ/~.clean_up.sql: prevent "column already exists" errors when there is an input column of the same name as an output column
bugfix: lib/runscripts/datasrc_dir.run: import(): don't run `sql/install` if the schema already exists, because this will try to rerun all the schema-creation queries. note that this idempotent functionality was not provided by the `make .../install` target that was previously used (idempotency is new with new-style import).
