Project

General

Profile

Activity

From 08/02/2013 to 08/31/2013

08/31/2013

07:47 PM Revision 10850: /README.TXT: Full database import: time to wait for the import to finish: updated to time in inputs/import.stats.xls
Aaron Marcuse-Kubitza
07:44 PM Revision 10849: bugfix: bin/import_all: `rm inputs/.TNRS/tnrs/tnrs.make.lock`: need to use `"rm"` instead of `rm` so that we don't use any rm alias the user might have in their shell (import_all is run in the calling shell so that the jobs are owned by the calling shell)
Aaron Marcuse-Kubitza
07:36 PM Revision 10848: bugfix: mappings/VegCore-VegBIEN.csv: don't map datasetURL to source.url for taxa-only data (this mapping should only occur for Source tables)
Aaron Marcuse-Kubitza
07:27 PM Revision 10847: bin/import_all: added step to remove any leftover TNRS lockfile (previously done manually)
Aaron Marcuse-Kubitza
06:46 PM Revision 10846: planning/timeline/timeline.2013.xls: updated for progress
Aaron Marcuse-Kubitza
06:32 PM Revision 10845: bugfix: lib/sql_io.py: put_table(): Getting output table pkeys of existing/inserted rows: need to include the index cond in the join condition here, too (using var join_custom_cond), so that an index scan can be used instead of a much slower full-table sort
Aaron Marcuse-Kubitza
06:01 PM Revision 10844: bugfix: schemas/vegbien.sql: locationevent: locationevent_unique_within_location unique index: added COALESCE(...) expression around location_id since it is nullable, and this is needed for the left and right sides of the join to exactly match up to use an index scan
Aaron Marcuse-Kubitza
05:52 PM Revision 10843: bugfix: lib/sql_io.py: put_table(): DuplicateKeyException: need to include any index cond in the join condition, so that an index scan can be used instead of a much slower full-table sort (otherwise the query planner will not know that it can restrict results to rows satisfying the index cond)
Aaron Marcuse-Kubitza
05:48 PM Revision 10842: lib/sql_gen.py: Join: added custom_cond param that can be used to add to the JOIN condition
Aaron Marcuse-Kubitza
01:02 AM Revision 10841: lib/sql.py: distinct_table(): support custom filters on the distincting query
Aaron Marcuse-Kubitza
01:01 AM Revision 10840: lib/sql_gen.py: ColValueCond: support conds that are just plain SQL (without separate left and right sides) using special custom_cond flag value
Aaron Marcuse-Kubitza

08/30/2013

11:18 PM Revision 10839: bugfix: inputs/input.Makefile: %/test: in by_col mode, also need to run %/test.by_col.xml
Aaron Marcuse-Kubitza
10:38 PM Revision 10838: lib/sql_io.py: ensure_cond(): documented meaning of passed, failed params (at least one row passed/failed the constraint)
Aaron Marcuse-Kubitza
06:28 PM Revision 10837: fix: web/links/index.htm: PostgreSQL: vacuuming: moved info about autovacuum process being aborted to correct bookmark
Aaron Marcuse-Kubitza
06:25 PM Revision 10836: web/links/index.htm: updated to Firefox bookmarks. PostgreSQL: added info on vacuuming and analyzing.
Aaron Marcuse-Kubitza
06:06 PM Revision 10835: lib/runscripts/util.run: usage: documented that this usage also applies to all files that include this file
Aaron Marcuse-Kubitza
06:06 PM Revision 10834: lib/runscripts/util.run: usage: clarified that the cmd to run is a function
Aaron Marcuse-Kubitza
06:03 PM Revision 10833: added schemas/postgresql*.conf.run, which installs these config files and takes care of restarting the server
Aaron Marcuse-Kubitza
06:02 PM Revision 10832: added lib/runscripts/pg.conf.run, which installs PostgreSQL config files
Aaron Marcuse-Kubitza
06:01 PM Revision 10831: added lib/runscripts/install.run, analogous to import.run
Aaron Marcuse-Kubitza
04:39 PM Revision 10830: fix: schemas/postgresql*.conf: turn on autovacuum logging (log_autovacuum_min_duration = 0) so we can verify if autovacuum is happening
Aaron Marcuse-Kubitza
04:35 PM Revision 10829: bugfix: lib/db_xml.py: put_table(): turned off db.autoanalyze, since forcing an ANALYZE after every bulk insert is inefficient for small datasources. the default autovacuum settings in schemas/postgresql.conf should be fine; however, the frequency and/or threshold may need to be increased if autovacuum does not ANALYZE frequently enough to replace db.autoanalyze.
Aaron Marcuse-Kubitza
03:37 PM Revision 10828: /run: taxon_trait/make(): order by scientificName, measurementType as described in the taxon_trait table comment
Aaron Marcuse-Kubitza
03:36 PM Revision 10827: lib/sh/db.sh: mk_select(): added support for ORDER BY
Aaron Marcuse-Kubitza
03:30 PM Revision 10826: /run: added taxon_trait/make()
Aaron Marcuse-Kubitza
03:28 PM Revision 10825: lib/sh/db.sh: added pg_export_table_to_dir(), analogous to pg_export_table_to_dir_no_header()
Aaron Marcuse-Kubitza
03:03 PM Revision 10824: schemas/vegbien.sql: taxon_trait: added query to use to export
Aaron Marcuse-Kubitza
02:52 PM Revision 10823: inputs/NVS/*/map.csv: Taxon Growth Form: mapped to VegBIEN.growthform enum, using http://www.fgdc.gov/standards/projects/FGDC-standards-projects/vegetation/NVCS_V2_FINAL_2008-02.pdf#page=83&section.page=76 . documented values used by each table.
Aaron Marcuse-Kubitza
02:18 PM Revision 10822: lib/sh/util.sh: is_array(): handle unset vars (=false). this fixes a bug in pg_export_table_no_header, which produced the error "lib/sh/util.sh: line 290: declare: cols: not found".
Aaron Marcuse-Kubitza
02:06 PM Revision 10821: fix: lib/sh/util.sh: join(): documented that delim must be a single char
Aaron Marcuse-Kubitza
07:15 AM Revision 10820: bugfix: /README.TXT: on a live machine, you should put the following in your .profile: need to make svn files web-accessible, because these are used by fs.vegpath.org links (such as to the ERD, etc.). note that this does not affect unversioned files, because these get the right permissions on the local machine instead (see Testing > On a development machine, you should put the following in your .profile).
Aaron Marcuse-Kubitza
07:07 AM Revision 10819: /README.TXT: to backup files not in Time Machine: added command to start the PostgreSQL server
Aaron Marcuse-Kubitza
06:58 AM Revision 10818: bugfix: /README.TXT: to synchronize a Mac's settings with my testing machine's: don't upload ~/.profile, etc. to jupiter because these files are different on each machine. they can instead be synced manually.
Aaron Marcuse-Kubitza
06:52 AM Revision 10817: /README.TXT: to backup files not in Time Machine: added command to stop the PostgreSQL server
Aaron Marcuse-Kubitza
06:49 AM Revision 10816: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: noted that ./fix_perms should be run on all machines
Aaron Marcuse-Kubitza
06:36 AM Revision 10815: removed unused dir analysis/ (originally requested by Jim)
Aaron Marcuse-Kubitza
06:31 AM Revision 10814: bugfix: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine:: added step to run `make backups/TNRS.backup/download live=1`, because bin/sync_upload does not sync this due to filters in backups/.rsync_filter.download
Aaron Marcuse-Kubitza
06:11 AM Revision 10813: /README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: added step to run ./fix_perms so that there are fewer permissions diffs to review
Aaron Marcuse-Kubitza
06:07 AM Revision 10812: bugfix: /README.TXT: to synchronize a Mac's settings with my testing machine's: upload: `(cd ~/Dropbox/svn/; svn up)`: use `up` instead so that the needed --force option is applied
Aaron Marcuse-Kubitza
05:37 AM Revision 10811: inputs/VegBank/*/postprocess.sql: added primary keys to derived tables
Aaron Marcuse-Kubitza
05:15 AM Revision 10810: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
05:11 AM Revision 10809: schemas/VegCore/VegCore.ERD.mwb: individual_observation.individual: documented that it is optional because an individual_observation cannot have an associated individual unless the individual is traceable to a specific plant
Aaron Marcuse-Kubitza
05:09 AM Revision 10808: schemas/VegCore/VegCore.ERD.mwb: individual_observation.place_observed_at: made it optional because some individual_observations (e.g. of the plant a specimen was collected from) may be missing location information. however, an individual_observation cannot have an associated individual unless the individual is traceable to a specific plant.
Aaron Marcuse-Kubitza
05:03 AM Revision 10807: schemas/VegCore/VegCore.ERD.mwb: specimen: added individual_observation, which stores observations about the plant the specimen was collected from. (some specimens may not be traceable to a reobservable individual, but will still have these plant observations.) specimen_observation: adjusted position to fully display the HAS-A connector to specimen.
Aaron Marcuse-Kubitza
03:44 AM Revision 10806: planning/timeline/timeline.2013.xls: updated for progress. rebalanced dots.
Aaron Marcuse-Kubitza
01:39 AM Revision 10805: planning/timeline/timeline.2013.xls: added separate task for Individual datasource refresh (separate from Individual datasource *removal*), because we also need to optimize the reload of datasources. the reload is most likely slow because rows are being added to very large tables.
Aaron Marcuse-Kubitza
01:21 AM Revision 10804: /README.TXT: Single datasource import: run commands in the background, since these are long-running commands
Aaron Marcuse-Kubitza
12:57 AM Revision 10803: planning/timeline/timeline.2013.xls: moved Attribution and conditions of use before Flatten the datasources as suggested in meeting with Mark
Aaron Marcuse-Kubitza
12:42 AM Revision 10802: schemas/vegbien.sql: datasource_rm(): runtime: added runtime of MO (55 min, 0.85 ms/row), which has a much larger # of rows than ACAD (4 million instead of 45,000). updated GBIF runtime estimate (~13 h) with more accurate ms/row from MO.
Aaron Marcuse-Kubitza

08/29/2013

11:19 PM Revision 10801: schemas/vegbien.sql: datasource_rm(): estimated runtime for GBIF (~10 h). note that this is still significantly shorter than the import time (3.4 days).
Aaron Marcuse-Kubitza
11:11 PM Revision 10800: schemas/vegbien.sql: datasource_rm(): documented how to calculate runtime
Aaron Marcuse-Kubitza
11:04 PM Revision 10799: schemas/vegbien.sql: datasource_rm(): documented runtime for ACAD: 30 s; 0.61 ms/row
Aaron Marcuse-Kubitza
05:09 PM Revision 10798: inputs/input.Makefile: rm: use new datasource_rm(), which encapsulates the schema-specific aspects of removing a datasource
Aaron Marcuse-Kubitza
04:59 PM Revision 10797: bugfix: schemas/vegbien.sql: datasource_rm(): set_config(): don't name the is_local param because it is not a named parameter
Aaron Marcuse-Kubitza
04:50 PM Revision 10796: schemas/vegbien.sql: added datasource_rm(). this uses an internal schema-scoping parameter to ensure that the function always operates on tables in the schema it was *defined* in, rather than tables in the search_path. this ensures that when the public schema is renamed (e.g. from an imported version), the function will continue to operate on its own schema rather than whichever schema happens to be called public. this avoids any surprises if you are trying to remove a datasource in one schema, and don't want it to unintentionally be removed in another schema instead.
Aaron Marcuse-Kubitza
04:20 PM Revision 10795: schemas/util.sql: added schema_ident()
Aaron Marcuse-Kubitza
03:20 PM Revision 10794: schemas/util.sql: added schema(regtype), schema(anyelement)
Aaron Marcuse-Kubitza
02:07 PM Revision 10793: inputs/.TNRS/schema.sql: added covering indexes on foreign keys where needed. this enables rows to be cascadingly deleted without a full table scan.
Aaron Marcuse-Kubitza
01:58 PM Revision 10792: schemas/vegbien.sql: added covering indexes on foreign keys where needed. this enables rows to be cascadingly deleted without a full table scan.
Aaron Marcuse-Kubitza

08/27/2013

11:11 PM Revision 10791: planning/timeline/timeline.2013.xls: increased font size for better readability at 100% (which is also the printed size). note that the timeline is normally zoomed in, so you don't see the actual font size.
Aaron Marcuse-Kubitza
10:52 PM Revision 10790: inputs/.TNRS/schema.sql: tnrs: instructions for when changing this table's schema: updated to use new `inputs/.TNRS/data.sql.run refresh`
Aaron Marcuse-Kubitza
10:50 PM Revision 10789: inputs/.TNRS/data.sql.run: added refresh() target which runs inputs/test_taxonomic_names/test_scrub
Aaron Marcuse-Kubitza
10:34 PM Revision 10788: inputs/test_taxonomic_names/test_scrub: added step to update inputs/.TNRS/data.sql to the now-refreshed TNRS sample data (this updating step is now automated)
Aaron Marcuse-Kubitza
10:32 PM Revision 10787: inputs/.TNRS/schema.sql: tnrs: updated steps to run when changing this table's schema, to use new TNRS editing workflow
Aaron Marcuse-Kubitza
10:14 PM Revision 10786: inputs/.TNRS/data.sql: re-ran TNRS using `inputs/test_taxonomic_names/test_scrub; rm=1 inputs/.TNRS/data.sql.run export_`
Aaron Marcuse-Kubitza
10:13 PM Revision 10785: /README.TXT: Full database import: fixing TNRS errors: noted that inputs/test_taxonomic_names/test_scrub re-runs TNRS
Aaron Marcuse-Kubitza
10:12 PM Revision 10784: /README.TXT: Full database import: fixing TNRS errors: updated instructions for new TNRS schema editing workflow
Aaron Marcuse-Kubitza
09:53 PM Revision 10783: inputs/.TNRS/data.sql: generate from the DB using `rm=1 inputs/.TNRS/data.sql.run export_` instead of being a hand-edited file
Aaron Marcuse-Kubitza
09:50 PM Revision 10782: added inputs/.TNRS/data.sql.run for syncing data.sql directly with the DB without needing to use inputs/test_taxonomic_names/test_scrub just to export the sample data. (however, when modifying the tnrs table, it may still be easier to generate new sample data using test_scrub rather than refactoring the table in place.)
Aaron Marcuse-Kubitza
09:35 PM Revision 10781: added lib/runscripts/data.pg.sql.run (analogous to schema.pg.sql.run for data-only SQL scripts)
Aaron Marcuse-Kubitza
09:32 PM Revision 10780: added lib/runscripts/file.pg.sql.run and use it in schema.pg.sql.run
Aaron Marcuse-Kubitza
09:25 PM Revision 10779: added lib/runscripts/schema.pg.sql.run and use it in inputs/.TNRS/schema.sql.run
Aaron Marcuse-Kubitza
09:18 PM Revision 10778: inputs/.TNRS/schema.sql: generate from the DB using `rm=1 inputs/.TNRS/schema.sql.run export_` instead of being a hand-edited file. this makes it much easier to edit the (now frequently-changing) TNRS schema directly in pgAdmin (which is graphical), rather than having to manually copy SQL changes from pgAdmin to the file.
Aaron Marcuse-Kubitza
09:15 PM Revision 10777: inputs/.TNRS/schema.sql.run: export_(): added usage
Aaron Marcuse-Kubitza
09:12 PM Revision 10776: added inputs/.TNRS/schema.sql.run, which syncs schema.sql with the DB
Aaron Marcuse-Kubitza
09:07 PM Revision 10775: bugfix: lib/sh/db.sh: pg_dump(): don't default $struct flag to on, because both structure and data should be printed by default
Aaron Marcuse-Kubitza
09:02 PM Revision 10774: lib/sh/db.sh: pg_dump(): added create_schema= flag to remove CREATE SCHEMA statements (useful if the schema already exists)
Aaron Marcuse-Kubitza
08:59 PM Revision 10773: bugfix: lib/sh/util.sh: set_fds(): remove empty redirects resulting from using `redirs= cmd...` to clear the redirs and then using $redirs as an array
Aaron Marcuse-Kubitza
08:47 PM Revision 10772: fix: lib/sh/util.sh: set_fds(): documented that it does not currently support redirecting an fd to itself (due to bash bugs that require the dest fd to be closed before it can be reopened)
Aaron Marcuse-Kubitza
08:44 PM Revision 10771: bugfix: lib/sh/util.sh: stdout2fd(): don't add >&$fd redirect if the fd is 1, because redir does not currently support redirecting an fd to itself (due to bash bugs that require the dest fd to be closed before it can be reopened)
Aaron Marcuse-Kubitza
08:40 PM Revision 10770: lib/sh/util.sh: filter_fd(): factored out >() subshell command into stdout2fd() for clarity
Aaron Marcuse-Kubitza
08:33 PM Revision 10769: bugfix: lib/sh/util.sh: redir(): unset redirs so that you don't redirect again in the invoked command
Aaron Marcuse-Kubitza
08:29 PM Revision 10768: fix: lib/sh/util.sh: filter_fd(): documented that ${redirs[@]} must not be set to an outer value
Aaron Marcuse-Kubitza
07:41 PM Revision 10767: fix: inputs/ARIZ/omoccurrences/map.csv: occurrenceID: remapped to EQUIV#to:occid instead of DUPLICATE#of:occid since these are not exact duplicates
Aaron Marcuse-Kubitza
07:30 PM Revision 10766: lib/runscripts/util.run: added to_top_file alias for use with $top_file
Aaron Marcuse-Kubitza
07:10 PM Revision 10765: lib/sh/local.sh: added pg_dump_local()
Aaron Marcuse-Kubitza
07:09 PM Revision 10764: lib/sh/db.sh: added pg_dump(), using the code in bin/pg_dump_vegbien with clarity improvements
Aaron Marcuse-Kubitza
07:06 PM Revision 10763: lib/sh/db.sh: added pg_cmd() (analogous to mysql_cmd() for PostgreSQL), and use it in psql(), so that other PostgreSQL operations can use this to set the PG* connection/login vars
Aaron Marcuse-Kubitza
05:36 PM Revision 10762: planning/timeline/timeline.2013.xls: updated dots for new priority order
Aaron Marcuse-Kubitza
05:33 PM Revision 10761: planning/timeline/timeline.2013.xls: moved optimization of individual datasource removal before flattening the datasources to a common schema as suggested in meeting with Mark
Aaron Marcuse-Kubitza
04:00 PM Revision 10760: lib/runscripts/datasrc_dir.run: include of import.run: use .rel instead of `. "$(dirname "${BASH_SOURCE[0]}")"/...`
Aaron Marcuse-Kubitza
03:59 PM Revision 10759: lib/runscripts/datasrc_dir.run: moved commands related to any runscript in the datasrc dir to new in_datasrc_dir.run
Aaron Marcuse-Kubitza
03:57 PM Revision 10758: inputs/*/Specimen/test.xml.ref with eventDate->dateCollected mappings: updated test outputs to match mapping
Aaron Marcuse-Kubitza
03:52 PM Revision 10757: some inputs/*/*/unmapped_terms.csv: updated now that datasetURL is mapped (this does not affect the mappings because it is only mapped for Source tables)
Aaron Marcuse-Kubitza
03:43 PM Revision 10756: bugfix: inputs/ARIZ/omoccurrences/map.csv: fixed one-to-many mapping for modified (created by the automapper?)
Aaron Marcuse-Kubitza
02:38 PM Revision 10755: lib/sh/db.sh: pg_export(): added usage
Aaron Marcuse-Kubitza
01:54 PM Revision 10754: inputs/.TNRS/schema.sql: moved source code comments to in-schema COMMENT ON comments so all the info in schema.sql is in the DB
Aaron Marcuse-Kubitza
01:47 PM Revision 10753: inputs/.TNRS/schema.sql: views that use * as the column list: added comments to indicate that this is the case, so that the views can be updated in place rather than only by reinstalling the TNRS schema
Aaron Marcuse-Kubitza
01:20 PM Revision 10752: updated backups/TNRS.backup.md5
Aaron Marcuse-Kubitza
01:19 PM Revision 10751: planning/timeline/timeline.2013.xls: clarified note about the purpose of the dots
Aaron Marcuse-Kubitza
01:03 PM Revision 10750: added backups/vegbien.r10548.backup.md5
Aaron Marcuse-Kubitza
01:03 PM Revision 10749: bugfix: backups/: svn:ignore: removed *.md5, which should be under version control
Aaron Marcuse-Kubitza
12:55 PM Revision 10748: inputs/input.Makefile: scrub: documented that using & (background process) ignores TNRS errors, so that TNRS bugs do not prevent the remaining tables from being imported even if TNRS can't be run
Aaron Marcuse-Kubitza
12:49 PM Revision 10747: inputs/.TNRS/schema.sql: tnrs: util.set_col_types() runtime: updated for most recent ALTER COLUMN TYPE command (9 min)
Aaron Marcuse-Kubitza
12:25 PM Revision 10746: inputs/.TNRS/schema.sql: tnrs.Time_submitted: renamed to batch and added fkey to batch.id. this requires including the batch table in inputs/.TNRS/data.sql, so that the fkey is satisfied (batch entries are already added by bin/tnrs_db.
Aaron Marcuse-Kubitza
11:42 AM Revision 10745: updated backups/TNRS.backup
Aaron Marcuse-Kubitza
11:38 AM Revision 10744: /README.TXT: Full database import: To back up DB (staging tables and last import) separately: added step to upload backups to jupiter
Aaron Marcuse-Kubitza
11:30 AM Revision 10743: /README.TXT: Full database import: To back up DB (staging tables and last import) separately: added step to remake backups/TNRS.backup
Aaron Marcuse-Kubitza

08/26/2013

08:45 PM Revision 10742: bin/tnrs_db: add entry to new batch table
Aaron Marcuse-Kubitza
07:48 PM Revision 10741: inputs/.TNRS/schema.sql: batch: reset name of id_by_time unique constraint since this field is now in the batch table
Aaron Marcuse-Kubitza
07:46 PM Revision 10740: inputs/.TNRS/schema.sql: download_settings: renamed to batch_download_settings because this table is actually specific to the batch, and it does not make sense to have a download settings file without a batch
Aaron Marcuse-Kubitza
07:32 PM Revision 10739: inputs/.TNRS/schema.sql: download_settings.id: added fkey to batch.id to create a 1:1 relationship with optional participation by download_settings. note that this relationship happens to be the same as SQL inheritance, as used in VegCore, but in this case, the 1:1 relationship is not related to inheritance.
Aaron Marcuse-Kubitza
06:30 PM Revision 10738: inputs/.TNRS/schema.sql: client_version: added table, column comments with info on how to retrieve each value
Aaron Marcuse-Kubitza
06:28 PM Revision 10737: inputs/.TNRS/schema.sql: added client_version table for svn revisions, with fkey from batch
Aaron Marcuse-Kubitza
06:23 PM Revision 10736: inputs/.TNRS/schema.sql: added batch table and moved download_settings.time_submitted, id_by_time to it since these are not related to the download_settings file
Aaron Marcuse-Kubitza
05:04 PM Revision 10735: fix: planning/timeline/timeline.2013.xls: Switching to new-style import: updated hyperlink
Aaron Marcuse-Kubitza
05:02 PM Revision 10734: planning/timeline/timeline.2013.xls: moved Individual datasource removal under Streamline process of mapping and adding a new datasource
Aaron Marcuse-Kubitza
04:49 PM Revision 10733: planning/timeline/timeline.2013.xls: added note that the purpose of the dots is to show what tasks should be worked on. in some cases, they are also proportional to the complexity of the task, but this may not be the case if e.g. a task was given different priorities in different months, or worked on in different amounts.
Aaron Marcuse-Kubitza
04:38 PM Revision 10732: fix: planning/timeline/timeline.2013.xls: matched supertask status to subtask status
Aaron Marcuse-Kubitza
04:35 PM Revision 10731: planning/timeline/timeline.2013.xls: made Switching to new-style import a subtask of Streamline process of mapping and adding a new datasource because new-style import automates many of the datasource-mapping tasks that previously needed to be done by hand
Aaron Marcuse-Kubitza
04:33 PM Revision 10730: planning/timeline/timeline.2013.xls: reordered for priorities and to-do assignments from last conference call (wiki.vegpath.org/2013-08-22_conference_call#Decisions-made)
Aaron Marcuse-Kubitza
04:32 PM Revision 10729: planning/timeline/timeline.2013.xls: updated for August progress and recently-added tasks
Aaron Marcuse-Kubitza
01:49 PM Revision 10728: inputs/.TNRS/schema.sql: added VegCore-style id column as the primary key, instead of using time_submitted directly. this enables always using the same name for the pkey. the pkey is now autopopulated from time_submitted in a trigger, using helper column id_by_time. the user is now also able to specify their own globally-unique ID that is not based on the time_submitted.
Aaron Marcuse-Kubitza

08/25/2013

11:22 PM Revision 10727: inputs/.TNRS/schema.sql: download_settings comment: changed name of button to Download settings, which had gotten auto-replaced to download_settings
Aaron Marcuse-Kubitza
11:08 PM Revision 10726: inputs/.TNRS/schema.sql: Download settings table: renamed to download_settings because although Download settings is the verbatim name of the button that this info comes from, it is not necessary to name the table a particular way in order to match up data to it correctly, so we can just use the standard naming convention (wiki.vegpath.org/u-name#format) and avoid the need to enclose the name in ""
Aaron Marcuse-Kubitza

08/24/2013

06:00 PM Revision 10725: inputs/.TNRS/schema.sql: added Download settings table, which stores data from http://tnrs.iplantcollaborative.org/TNRSapp.html > Submit List > results section > Download settings > settings.txt
Aaron Marcuse-Kubitza
04:07 PM Revision 10724: inputs/.TNRS/Source/map.csv: mapped datasetURL
Aaron Marcuse-Kubitza

08/23/2013

11:43 PM Revision 10723: inputs/.geoscrub/Source/map.csv: mapped datasetURL
Aaron Marcuse-Kubitza
11:41 PM Revision 10722: mappings/VegCore-VegBIEN.csv: mapped datasetURL
Aaron Marcuse-Kubitza
11:38 PM Revision 10721: mappings/VegCore-VegBIEN.csv: mapped datasetURL
Aaron Marcuse-Kubitza

08/22/2013

06:12 PM Revision 10720: fix: mappings/VegCore-VegBIEN.csv: source__modified_date: remapped to pubdate instead of datelastmodified because this is actually metadata for the source itself, rather than for the VegBIEN record *of* the source
Aaron Marcuse-Kubitza
05:56 PM Revision 10719: fix: inputs/.geoscrub/Source/map.csv: source__modified_date: use the mtime of the CSV file instead, since this is closer to the actual version of the biengeo code at the time it was run
Aaron Marcuse-Kubitza
05:41 PM Revision 10718: inputs/.geoscrub/Source/map.csv: mapped source__modified_date. note that the test must be run with inputs/.geoscrub/Source/run instead of `make inputs/.geoscrub/Source/test` to add these metadata columns to the staging table.
Aaron Marcuse-Kubitza
05:38 PM Revision 10717: mappings/VegCore-VegBIEN.csv: mapped source__modified_date (different from vegcore.vegpath.org?modified, which is for the data record)
Aaron Marcuse-Kubitza
05:36 PM Revision 10716: mappings/VegCore.htm: regenerated from wiki. added source__version (= edition), source__modified_date.
Aaron Marcuse-Kubitza
05:33 PM Revision 10715: bugfix: schemas/util.sql: set_col_names_with_metadata(): *rename* any metadata cols rather than re-adding them with new names
Aaron Marcuse-Kubitza
04:38 PM Revision 10714: mappings/VegCore-VegBIEN.csv: mapped edition
Aaron Marcuse-Kubitza
04:36 PM Revision 10713: bugfix: inputs/.geoscrub/{Source,geoscrub_output}/VegBIEN.csv: switched to the version needed for new-style datasources
Aaron Marcuse-Kubitza
04:12 PM Revision 10712: inputs/.geoscrub/Source/map.csv: mapped edition (the version), using `svn info derived/biengeo/`
Aaron Marcuse-Kubitza
03:53 PM Revision 10711: schemas/vegbien.sql: source.revision: renamed to import_revision for clarity
Aaron Marcuse-Kubitza
03:52 PM Revision 10710: schemas/vegbien.my.sql: updated with `make schemas/remake`
Aaron Marcuse-Kubitza
03:44 PM Revision 10709: schemas/vegbien.sql: source: datecreated, datelastmodified: default to now() like in VegBank (schemas/VegBank/vegbank.sql)
Aaron Marcuse-Kubitza
03:29 PM Revision 10708: schemas/vegbien.sql: source: added datecreated, datelastmodified, etc. for source-level tracking of import and revision (wiki.vegpath.org/2013-08-22_conference_call#source-level-tracking-of-import-and-revision)
Aaron Marcuse-Kubitza
02:54 PM Revision 10707: added derived/biengeo/ from https://projects.nceas.ucsb.edu/nceas/projects/biengeo/repository/
Aaron Marcuse-Kubitza
02:50 PM Revision 10706: added /derived
Aaron Marcuse-Kubitza
01:04 PM Revision 10705: web/links/index.htm: updated to Firefox bookmarks. Gmvault: added steps to do full backup and to backup only new e-mails
Aaron Marcuse-Kubitza
12:48 PM Revision 10704: planning/timeline/timeline.2013.xls: flagged timeline issues that can be done by iPlant personnel: Attribution and conditions of use, Geoscrubbing re-run, Geoscrubbing automated pipeline, Improve and complete data provider metadata, Obtain any additional new data
Aaron Marcuse-Kubitza
11:12 AM Revision 10703: web/links/index.htm: updated to Firefox bookmarks. Gmvault: added run instructions for Mac.
Aaron Marcuse-Kubitza
10:50 AM Revision 10702: web/links/index.htm: updated to Firefox bookmarks. added link to Gmvault (Gmail backup), which wouldn't install for me on Mac (but that may be because I'm using 10.8, and Gmvault is for 10.7/10.6)
Aaron Marcuse-Kubitza
10:26 AM Revision 10701: added planning/meetings/BIEN conference call availability.xlsx (backup of Google spreadsheet)
Aaron Marcuse-Kubitza
10:10 AM Revision 10700: planning/timeline/timeline.2013.xls: updated for changes made in the conference call: moved Data provider validations (spot-checking) to beginning since that seems to have been decided to be a higher priority than architectural changes
Aaron Marcuse-Kubitza

08/21/2013

06:50 PM Revision 10699: schemas/util.sql: combining functions taking anyelement params which could be text: take text param instead, so that other argument types (e.g. integer) will first be implicitly cast to text instead of trying to concatenate integers directly. this fixes a bug in the VegBank.stemcount_,stemlocation_ _join() of two integer pkeys, which first needed to be cast to text. anyelement was previously used so that other text-like types such as varchar could also be used, but varchar is implicitly castable to text so keeping anyelement should not be necessary.
Aaron Marcuse-Kubitza
06:07 PM Revision 10698: planning/timeline/timeline.2013.xls: added tasks to Avoid DB restructuring when ingesting a new datasource and Streamline process of mapping and adding a new datasource (not yet put in priority order)
Aaron Marcuse-Kubitza

08/20/2013

08:49 PM Revision 10697: inputs/VegBank/observation_/test.xml.ref: updated inserted row count
Aaron Marcuse-Kubitza
02:40 PM Revision 10696: bugfix: inputs/VegBank/stemlocation_/map.csv: also _join together taxonimportance_id, stemcount_id for aggregateOrganismObservationID so that the aggregateoccurrence pkeys match up with those imported from stemcount_
Aaron Marcuse-Kubitza
01:40 PM Revision 10695: bugfix: inputs/VegBank/stemcount_/map.csv: aggregateOrganismObservationID: prepend taxonimportance_id so that rows with only a taxonimportance entry (no stemcounts) will also have the required sourceaccessioncode
Aaron Marcuse-Kubitza
12:59 PM Revision 10694: schemas/vegbien.sql: analytical_stem: synced with analytical_stem_view using sync_analytical_stem_to_view()
Aaron Marcuse-Kubitza
12:58 PM Revision 10693: bugfix: schemas/vegbien.sql: sync_analytical_stem_to_view(): added re-creation of range_modeling_input view
Aaron Marcuse-Kubitza
11:40 AM Revision 10692: schemas/vegbien.sql: analytical_stem_view: added aggregateOrganismObservationID (aggregateoccurrence.sourceaccessioncode) so aggregateoccurrences can be matched back up to their input rows (e.g. VegBank.stemcount)
Aaron Marcuse-Kubitza
10:31 AM Revision 10691: inputs/VegBank/taxonobservation_/map.csv: plantname: remapped to DUPLICATE#of:plantconcept_plantname because this is an exact duplicate
Aaron Marcuse-Kubitza
10:24 AM Revision 10690: bugfix: inputs/VegBank/taxonobservation_/map.csv: updated input column names for renamings in inputs/VegBank/vegbank.~.clean_up.sql
Aaron Marcuse-Kubitza
10:21 AM Revision 10689: inputs/VegBank/taxonobservation_/map.csv: Species and lower ranks: remapped to EQUIV#to:plantname because these contain the taxonomic name at specific ranks, but plantname contains the taxonomic name of the plant itself, which is longer and populated more often
Aaron Marcuse-Kubitza

08/19/2013

05:48 AM Revision 10688: inputs/VegBank/taxonobservation_/create.sql: also join to plantname, since plantconcept.plantname may not always be populated when plantname.plantname is
Aaron Marcuse-Kubitza
05:36 AM Revision 10687: fix: inputs/VegBank/taxonobservation_/map.csv: Species and below: remapped to _alts of scientificName, because these are actually the *full* taxonomic name at that rank, not just the epithet. Genus: documented that it includes the genus author.
Aaron Marcuse-Kubitza
05:28 AM Revision 10686: inputs/VegBank/vegbank.~.clean_up.sql: disambiguated plantconcept.plantname, plantname.reference_id to enable joining plantconcept_->plantname
Aaron Marcuse-Kubitza
05:13 AM Revision 10685: fix: inputs/VegBank/taxonobservation_/map.csv: also mapped plantname to scientificName, since int_currplantscifull is not always provided when this is. (it cannot replace int_currplantscifull, because when int_currplantscifull also provided, this often leaves out lower ranks.) this should fill in taxonomic information for taxonobservations that are currently missing it.
Aaron Marcuse-Kubitza
03:27 AM Revision 10684: bugfix: schemas/vegbien.sql: analytical_stem_view: coordinates: use the coordinates from datasource_place instead of canon_place, because canon_place's coordinates are only what the geoscrubbing output and do not contain datasource-specific information such as coordsaccuracy_m
Aaron Marcuse-Kubitza
01:53 AM Revision 10683: inputs/VegBank/taxonobservation_/map.csv: collector_id: remapped to UNUSED. removed LEFT JOIN collector_id->party since this field is never populated.
Aaron Marcuse-Kubitza
01:16 AM Revision 10682: inputs/VegBank/plot_/map.csv: area|country|territory, region|state|province (from place table): remapped to DUPLICATE, since these have the same data as, and are populated less often than, their country/stateprovince couterparts
Aaron Marcuse-Kubitza

08/18/2013

11:39 PM Revision 10681: bugfix: inputs/VegBank/plot_/create.sql: need to join place.*plot_id* to plot.plot_id instead of plotplace_id. this is the cause of the "State is wrong, not Wyoming, but Tennessee" and "County is incorrect (not Powell, but Orange)" bugs in the VegBank spot-checking (wiki.vegpath.org/Spot-checking#Great-Smoky-Mountains-National-Park).
Aaron Marcuse-Kubitza
10:42 PM Revision 10680: inputs/VegBank/taxonobservation_/map.csv: int_origplantscifull: remapped to EQUIV (to authorplantname). this is the scrubbed originalScientificName, but we do our own scrubbing.
Aaron Marcuse-Kubitza
10:23 PM Revision 10679: fix: inputs/VegBank/taxonobservation_/map.csv: authorplantname: remapped to originalScientificName because it includes the name author
Aaron Marcuse-Kubitza
10:21 PM Revision 10678: inputs/VegBank/taxonobservation_/map.csv: mapped int_origplant*, int_currplant* to *scientificName/*taxonName/etc.
Aaron Marcuse-Kubitza
09:48 PM Revision 10677: inputs/VegBank/plot_/map.csv: elevation: documented that it has only 5 decimal places of precision, with only 9s and random #s after that
Aaron Marcuse-Kubitza
09:47 PM Revision 10676: inputs/VegBank/plot_/test.xml.ref: update rowcount
Aaron Marcuse-Kubitza
09:15 PM Revision 10675: inputs/VegBank/stemlocation_/map.csv: stemcode, stemxposition, stemyposition: remapped to UNUSED. stemhealth is the only data field in this table that is populated, which means that VegBank does not provide data on reobservable stems even though the schema supports it.
Aaron Marcuse-Kubitza
09:12 PM Revision 10674: inputs/VegBank/observation_/postprocess.sql: added pkey
Aaron Marcuse-Kubitza
08:19 PM Revision 10673: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
08:18 PM Revision 10672: schemas/VegCore/ERD/index.htm: regenerated
Aaron Marcuse-Kubitza
08:05 PM Revision 10671: schemas/VegCore/VegCore.ERD.mwb: taxon_observation: added observation_in_parent_place, which points to the observation of the same taxon/individual in the parent place. this accounts for VegBank allowing multiple taxonImportances per taxonObservation, which only makes sense when each taxonImportance is from a different stratum and they point to a common taxonObservation for the parent plot.
Aaron Marcuse-Kubitza
07:58 PM Revision 10670: bugfix: web/.htaccess: auto-detect dotpath in query string: need to include the path ($0) in the replacement, to avoid reverting to the root dir. (mod_rewrite replacements are not like relative URLs, which would interpret ?... as being relative to the *current* path, not the root.)
Aaron Marcuse-Kubitza
07:56 PM Revision 10669: bugfix: web/.htaccess: auto-detect dotpath in query string: added missing $ at end of regexp
Aaron Marcuse-Kubitza

08/10/2013

11:55 PM Revision 10668: inputs/VegBIEN/db/.htaccess: allow access to schemas other than the public schema by requiring a schema before the table. currently, this does not break any existing URLs.
Aaron Marcuse-Kubitza

08/09/2013

01:57 PM Revision 10667: planning/timeline/timeline.2013.xls: rebalanced between the months. Geoscrubbing re-run: extended to September since it's not clear when Jim will have time for this.
Aaron Marcuse-Kubitza
01:53 PM Revision 10666: planning/timeline/timeline.2013.xls: split Data provider validations (spot-checking) into separate tasks for VegBIEN and VegCore
Aaron Marcuse-Kubitza
01:49 PM Revision 10665: planning/timeline/timeline.2013.xls: updated for July progress
Aaron Marcuse-Kubitza
01:25 PM Revision 10664: schemas/VegCore/VegCore.ERD.mwb: geovalidation: added step at end to georeference all places to corrected_geoplace
Aaron Marcuse-Kubitza
12:51 PM Revision 10663: schemas/VegCore/VegCore.ERD.mwb: moved all required fields before optional fields
Aaron Marcuse-Kubitza
12:39 PM Revision 10662: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
12:23 PM Revision 10661: schemas/VegCore/VegCore.ERD.mwb: individual: identifying_place: documented that for specimens, this is the place the specimen was collected at. within_place: documented that this is not specified for specimens since their coordinates are usually not precise enough to identify an individual. specimen: documented that when there are multiple specimen replicates (copies) of a specimen, each gets its own specimen_observation pointing to the same specimen.
Aaron Marcuse-Kubitza
12:12 PM Revision 10660: schemas/VegCore/VegCore.ERD.mwb: individual: added identifying_place since some individuals may be identified by their coordinates rather than a tag
Aaron Marcuse-Kubitza
12:06 PM Revision 10659: schemas/VegCore/VegCore.ERD.mwb: added back aggregate_observation as a superclass of taxon_absence, taxon_presence. have only aggregate_observation, not taxon_observation, inherit from taxa_sampling_event because taxa_sampling_event is designed for multiple or aggregate samplings, but taxon_observation can be for just one occurrence. taxa_sampling_event: added required within_place since taxa must be sampled within a particular place. individual.place: renamed to within_place to clarify that each individual does not necessarily need its own place instance. individual_observation: added place_observed_at to serve the role that individual.place previously did (the place of an individual may change between observations).
Aaron Marcuse-Kubitza
10:41 AM Revision 10658: schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: renamed to individual_count for clarity. added count field. renamed traits to aggregating_traits since these are used in uniquely identifying the individual_count. taxon_presence: added cover_percent.
Aaron Marcuse-Kubitza
10:17 AM Revision 10657: schemas/VegCore/VegCore.ERD.mwb: taxon_presence: documented that this is actually equivalent to VegX's aggregateOrganismObservation ("An observation applying to all occurrences of an organism"), in spite of VegCore having its own aggregate_observation table with a different purpose
Aaron Marcuse-Kubitza
10:12 AM Revision 10656: schemas/VegX/.htaccess, TCS/.htaccess: prepend / to subpath so that it becomes an absolute XPath. the subpath is now usually provided as a dotpath translated to a dirpath, which would not have a leading / .
Aaron Marcuse-Kubitza
12:08 AM Revision 10655: fix: schemas/VegCore/VegCore.ERD.mwb.run: comment about why the Mac version does not export the PDF correctly: clarified that it is the *single-page* PDF where the lines take very long to load, and the *printed* PDF where the colors are wrong in Adobe Reader
Aaron Marcuse-Kubitza
12:02 AM Revision 10654: schemas/VegCore/VegCore.ERD.mwb.run: documented that to edit VegCore.ERD.mwb, you *must* use the Linux version of MySQL Workbench. this is because the Mac version does not export the PDF correctly: the colors are wrong in Adobe Reader and the lines take very long to load. unfortunately, you also cannot switch back and forth between the two versions, because they render the lines with a different spacing, which makes the tables different sizes and misaligns the connector lines. it is especially unfortunate that the Mac version messes up the PDFs, because it uses a smaller line spacing and therefore smaller tables which allows the tables to be spaced farther apart for better readability.
Aaron Marcuse-Kubitza

08/08/2013

11:49 PM Revision 10653: schemas/VegCore/VegCore.ERD.mwb: aggregate_observation- and stratum-related tables: documented VegBank equivalents
Aaron Marcuse-Kubitza
10:20 PM Revision 10652: schemas/VegCore/VegCore.my.sql: regenerated with new schema-qualifier statements
Aaron Marcuse-Kubitza
10:16 PM Revision 10651: schemas/VegCore/VegCore.my.sql: regenerated with new schema-qualifier statements
Aaron Marcuse-Kubitza
10:14 PM Revision 10650: bugfix: schemas/VegCore/VegCore.ERD.mwb.run: VegCore.my.sql/make(): use new create_db=1 option to mysqldump_local so that the dumpfile contains the necessary schema-qualifiers to be syncable with the ERD (i.e. to allow editing the ERD by changing the dumpfile)
Aaron Marcuse-Kubitza
07:15 PM Revision 10649: lib/sh/db.sh: mysqldump(): added create_db=1 flag to print the CREATE DATABASE statement
Aaron Marcuse-Kubitza
07:15 PM Revision 10648: bugfix: lib/sh/util.sh: pf(): echo func decl to stderr instead of stdout
Aaron Marcuse-Kubitza
07:10 PM Revision 10647: lib/sh/util.sh: echo_vars(): documented that it only prints vars that are defined
Aaron Marcuse-Kubitza
07:10 PM Revision 10646: lib/sh/util.sh: local_inv: also echo_vars the new var
Aaron Marcuse-Kubitza
06:17 PM Revision 10645: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
06:16 PM Revision 10644: schemas/VegCore/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
06:06 PM Revision 10643: schemas/VegCore/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
06:04 PM Revision 10642: bugfix: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: updated name of sampling_event to taxa_sampling_event
Aaron Marcuse-Kubitza
06:03 PM Revision 10641: schemas/VegCore/ERD/VegCore.ERD.png.map.htm: moved place_observation-related tables before sampling_event because plots-related tables should come before occurrence-related tables
Aaron Marcuse-Kubitza
06:01 PM Revision 10640: schemas/VegCore/VegCore.ERD.mwb: regenerated exports and udpated image map
Aaron Marcuse-Kubitza
05:34 PM Revision 10639: schemas/VegCore/VegCore.ERD.mwb: soil_observation: allow multiple soil_observations per place_observation (e.g. at different spots), as VegBank does
Aaron Marcuse-Kubitza
05:27 PM Revision 10638: schemas/VegCore/VegCore.ERD.mwb: method: spaced more evenly from other tables
Aaron Marcuse-Kubitza
05:21 PM Revision 10637: schemas/VegCore/VegCore.ERD.mwb: fixed lines
Aaron Marcuse-Kubitza
05:20 PM Revision 10636: schemas/VegCore/VegCore.ERD.mwb: *.footprint_geom_WKT: renamed to boundary_WKT for clarity
Aaron Marcuse-Kubitza
05:14 PM Revision 10635: schemas/VegCore/VegCore.ERD.mwb: added reobservable_presence, which allows a vouchered taxon_presence (e.g. a set of aggregate_observations) to have taxon redeterminations
Aaron Marcuse-Kubitza
04:44 PM Revision 10634: schemas/VegCore/VegCore.ERD.mwb: aggregate_observation: changed relationship to taxon_presence to many:1 instead of 1:1 inheritance, because there can be multiple aggregate_observations (e.g. size classes) for each aggregated taxon. added optional size_class field to indicate the aggregating group.
Aaron Marcuse-Kubitza
04:34 PM Revision 10633: bugfix: schemas/VegCore/VegCore.ERD.mwb: method: made parent optional because top-level methods will not have a parent. rearranged to make more room for place_observation, soil_observation.
Aaron Marcuse-Kubitza
04:18 PM Revision 10632: schemas/VegCore/VegCore.ERD.mwb: sampling_event: renamed to taxa_sampling_event for clarity. (fkey order has reversed.)
Aaron Marcuse-Kubitza
04:12 PM Revision 10631: schemas/VegCore/VegCore.ERD.mwb: place_observation: don't inherit from sampling_event because sampling_event refers specifically to sampling taxa, and does not have anything to do with environmental measurements on the place itself. size_class: moved to occurrence half since it relates specifically to aggregate_observations, rather than plot strata.
Aaron Marcuse-Kubitza
03:14 PM Revision 10630: schemas/VegCore/VegCore.ERD.mwb: stratum: added layer, size_class subclasses (with name now only required for layers)
Aaron Marcuse-Kubitza
02:51 PM Revision 10629: schemas/VegCore/VegCore.ERD.mwb: stratum_event: added fkey to parent sampling_event. stratum: made space for subclasses.
Aaron Marcuse-Kubitza
02:22 PM Revision 10628: schemas/VegCore/VegCore.ERD.mwb: added stratum_event. (the fkey order also got reversed.)
Aaron Marcuse-Kubitza
01:26 PM Revision 10627: bugfix: inputs/VegBank/observation_/create.sql: ensure only one row per observation by selecting the first soilobs for each observation
Aaron Marcuse-Kubitza
12:09 PM Revision 10626: bugfix: inputs/VegBank/stemlocation_/create.sql: include 1 row for each stemlocation rather than for each stemcount, because stemcounts are instead imported as part of stemcount_ (which also includes taxonimportance)
Aaron Marcuse-Kubitza

08/07/2013

04:51 PM Revision 10625: inputs/TEX/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
Aaron Marcuse-Kubitza
04:44 PM Revision 10624: inputs/TEX/Specimen2/: translated new multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
Aaron Marcuse-Kubitza
04:41 PM Revision 10623: bugfix: inputs/TEX/Specimen2/map.csv: DETERMINER FIRST NAME/LAST NAME: combined using _join_words() instead of the default _alt()
Aaron Marcuse-Kubitza
04:20 PM Revision 10622: inputs/TEX/Specimen2/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
Aaron Marcuse-Kubitza
04:13 PM Revision 10621: inputs/TEX/Specimen2/map.csv: MISC NOTES: remapped to locationRemarks since these primarily relate to the coords. PLANT DATA: remapped to occurrenceRemarks since this is the specimenDescription.
Aaron Marcuse-Kubitza
03:15 PM Revision 10620: inputs/TEX/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
Aaron Marcuse-Kubitza
10:22 AM Revision 10619: inputs/UNCC/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
Aaron Marcuse-Kubitza
09:50 AM Revision 10618: inputs/UNCC/Specimen/: translated multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns. reproductiveCondition: use the suggested values at DwC.vegpath.org?reproductiveCondition instead of just joining together the values in flower and fruit.
Aaron Marcuse-Kubitza
09:19 AM Revision 10617: inputs/UNCC/Specimen/map.csv: usdaRank, infrarank: documented that they should actually be mapped to infraspecificRank*. it is better to map them to taxonRank for now because infraspecificRank is now mapped to VegBIEN.
Aaron Marcuse-Kubitza
09:07 AM Revision 10616: mappings/VegCore.htm: regenerated from wiki. added EQUIV (also mapped in mappings/VegCore-VegBIEN.csv).
Aaron Marcuse-Kubitza
09:04 AM Revision 10615: inputs/UNCC/Specimen/map.csv: infrarank: remapped to EQUIV (of usdaRank). usdaRank, infrarank: documented the VegCore term that each would map to.
Aaron Marcuse-Kubitza
08:49 AM Revision 10614: inputs/UNCC/Specimen/postprocess.sql: infrarank: remove mismapped values "L." (belong in authors, and are already there)
Aaron Marcuse-Kubitza
08:30 AM Revision 10613: schemas/util.sql: added ?*>= operator (is populated more often than) from wiki.vegpath.org/Postgres_queries#is-populated-more-often-than
Aaron Marcuse-Kubitza
08:12 AM Revision 10612: bugfix: web/.htaccess: parse dotpath in query string: make dotpath replacement start with ./ instead of / (which would incorrectly create an absolute URL path). this bug only appears when there is a trailing /, because this causes the matched text to be empty instead of a filesystem path with leading /.
Aaron Marcuse-Kubitza
08:04 AM Revision 10611: bugfix: web/.htaccess: parse dotpath in query string: redirect so REQUEST_URI is populated from REQUEST_FILENAME: need to use - instead of $0 as the redirect destination so that REQUEST_FILENAME is used instead of the empty REQUEST_URI (i.e. the redirect vars are left untouched)
Aaron Marcuse-Kubitza
07:45 AM Revision 10610: bugfix: web/.htaccess: parse dotpath in query string: need to add trailing / if dotpath so that dest path is URL path, not filesystem path (when trailing / missing, matched path is filesystem path)
Aaron Marcuse-Kubitza
07:44 AM Revision 10609: backups/TNRS.backup.md5: updated for last backup
Aaron Marcuse-Kubitza
06:33 AM Revision 10608: schemas/util.sql: hstore(text[], anyelement): replaced with just hstore(text[], text) so that bare literals (of type unknown) are auto-cast to text instead of causing a "could not determine polymorphic type because input has type "unknown"" error
Aaron Marcuse-Kubitza
05:56 AM Revision 10607: inputs/UNCC/Specimen/map.csv: loanto, inorout: removed mapping to occurrenceRemarks because this information does not relate to the occurrence. it is also rarely populated.
Aaron Marcuse-Kubitza

08/06/2013

05:07 PM Revision 10606: inputs/import.stats.xls: added backup MD5 test time for last import
Aaron Marcuse-Kubitza
05:03 PM Revision 10605: inputs/import.stats.xls: added backup upload time for last import
Aaron Marcuse-Kubitza
04:55 PM Revision 10604: inputs/UNCC/Specimen/: translated single-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
Aaron Marcuse-Kubitza
04:53 PM Revision 10603: /README.TXT: Full database import: min disk space: updated import schema size for last import
Aaron Marcuse-Kubitza
04:30 PM Revision 10602: inputs/import.stats.xls: added backup times from last import
Aaron Marcuse-Kubitza
01:35 AM Revision 10601: inputs/import.stats.xls: Updated import times
Aaron Marcuse-Kubitza
01:16 AM Revision 10600: /README.TXT: Full database import: tailing inputs/analytical_db/logs/make_analytical_db.log.sql: increased # lines to 150 to include all lines for the last run
Aaron Marcuse-Kubitza
12:32 AM Revision 10599: bugfix: bin/import_times: filtering out the Source subdirs: need to match 1 at the *beginning* of the line only
Aaron Marcuse-Kubitza
12:29 AM Revision 10598: bin/import_times: filter out the Source subdirs, which now have single-row data and therefore are included in the rowcounts list
Aaron Marcuse-Kubitza

08/04/2013

06:07 PM Revision 10597: inputs/U/: switched to new-style import, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource
Aaron Marcuse-Kubitza
06:06 PM Revision 10596: bugfix: schemas/util.sql: reset_col_names(): remove metadata value columns from the map table entirely, so that they are not considered to be in the table (old-style import does not use metadata value columns, so there would be no column or a different column in the metadata value column's slot)
Aaron Marcuse-Kubitza
05:58 PM Revision 10595: bugfix: schemas/util.sql: trim(): need to remove *all* added columns, not just derived columns, because metadata value columns are also added by new-style import and need to be removed to restore a staging table to its pre-new-style state
Aaron Marcuse-Kubitza
05:55 PM Revision 10594: schemas/util.sql: added added_cols()
Aaron Marcuse-Kubitza
04:06 PM Revision 10593: bugfix: schemas/util.sql: reset_col_names(): need to avoid renaming metadata value columns, because the "from" entry is a value, not a column name
Aaron Marcuse-Kubitza
01:29 AM Revision 10592: inputs/U/Specimen/map.csv: translated remaining multi-column filters to postprocessing derived columns, using the steps at http://wiki.vegpath.org/Adding_new-style_import_to_a_datasource#Translating-filters-to-postprocessing-derived-columns
Aaron Marcuse-Kubitza
01:23 AM Revision 10591: inputs/CVS/stemLocation_/test.xml.ref: set inserted row count back. it had changed because $version was still set in the environment, and this was causing a non-emtpty public schema to be used as the testing schema.
Aaron Marcuse-Kubitza
01:10 AM Revision 10590: inputs/CVS/stemLocation_/test.xml.ref: updated inserted row count
Aaron Marcuse-Kubitza
12:59 AM Revision 10589: bin/after_import: usage: documented that it requires $version
Aaron Marcuse-Kubitza
12:58 AM Revision 10588: bugfix: /README.TXT: Full database import: To restart an aborted import for a specific table: bin/after_import: need to run it in the background
Aaron Marcuse-Kubitza
12:57 AM Revision 10587: /README.TXT: Full database import: To restart an aborted import for a specific table: added step to run bin/after_import
Aaron Marcuse-Kubitza

08/03/2013

09:14 PM Revision 10586: bin/import_all: use new bin/after_import
Aaron Marcuse-Kubitza
09:13 PM Revision 10585: added bin/after_import, which performs post-normalized-import actions separately from bin/import_all
Aaron Marcuse-Kubitza
03:34 PM Revision 10584: bugfix: /README.TXT: Full database import: To restart an aborted import for a specific table: added by_col=1
Aaron Marcuse-Kubitza
03:32 PM Revision 10583: /README.TXT: Full database import: added steps to restart an aborted import for a specific table
Aaron Marcuse-Kubitza
03:28 PM Revision 10582: inputs/input.Makefile: $(import): support restarting the import where it left off by setting continue=1. this is done by grepping the restart row out of the log file's last partition.
Aaron Marcuse-Kubitza
03:11 PM Revision 10581: inputs/input.Makefile: added %/import_scrub, similar to import_scrub but just imports one table
Aaron Marcuse-Kubitza
12:25 AM Revision 10580: bin/import_all: with_all import_scrub: documented that this step uses $by_col, so that users know to include by_col=1 when running this step separately
Aaron Marcuse-Kubitza
12:24 AM Revision 10579: bin/import_all: use column-based import (by_col=1) by default, instead of requiring the user to explicitly specify it. instead turn it off explicitly (by_col=) for row-based import.
Aaron Marcuse-Kubitza
12:03 AM Revision 10578: bugfix: /README.TXT: Full database import: To back up DB: after renaming current import to public: say to replace $version with the appropriate revision, because the $version env var should not be set (otherwise the backup will try to use a nonexistent import with the given revision #)
Aaron Marcuse-Kubitza
12:00 AM Revision 10577: /README.TXT: Full database import: To back up DB: updated instructions to inline setting of $dump_opts, like in bin/import_all
Aaron Marcuse-Kubitza

08/02/2013

11:55 PM Revision 10576: bin/import_all: don't set $dump_opts until running the backup command that uses it, so that the user can run this backup command separately just by copying the line out of the script (without worrying about env vars that need to be set, other than $version which is visible outside the script)
Aaron Marcuse-Kubitza
 

Also available in: Atom