Activity
From 12/20/2011 to 01/18/2012
01/18/2012
- 05:57 PM Revision 529: vegbien.sql: Removed no longer needed taxondetermination.plantname_id
- 05:53 PM Revision 528: VegBIEN: Made taxondetermination.plantconcept_id required and plantname_id optional
- 04:28 PM Revision 527: vegbien.sql: Added specimen.taxonoccurrence_id fkey
- 03:47 PM Revision 526: vegbien.sql: Removed no longer needed taxondetermination.individualplant_id, specimen_id fields
- 01:57 PM Revision 525: VegBIEN: Renamed plot to location to reflect that a location can also be used for specimens
- 01:29 PM Revision 524: vegbien.sql: Added individualplant.stemcount
- 12:38 PM Revision 523: vegbien.sql: Added accessioncode fields to existing, modified tables
- 12:21 PM Revision 522: vegbien.sql: Added accessioncode fields to new tables
01/17/2012
- 04:56 PM Revision 521: vegbien.sql: Removed no longer needed individualplant.taxonarea
- 04:40 PM Revision 520: VegBIEN: Moved taxondetermination.sourceaccessionnumber and museum_id to specimen
- 03:32 PM Revision 519: vegbien.sql: Split taxondetermination UNIQUE CONSTRAINT into one constraint for each main key (taxonoccurrence_id, specimen_id, individualplant_id) to deal with Postgres treating a row with NULL in any of those columns as distinct. Added CHECK constraint to require one of taxonoccurrence_id, specimen_id, individualplant_id to be NOT NULL.
- 02:47 PM Revision 518: VegBIEN: Renamed museumaccessionnumber to sourceaccessionnumber
- 02:42 PM Revision 517: VegBIEN: Renamed authorcode to sourceid
- 02:33 PM Revision 516: vegbien.sql: Updated sizeclass to use ranges for parameters and remove taxonarea
01/13/2012
- 05:25 PM Revision 515: input.Makefile: Fixed verify by making %.out: % rule use %.make instead
- 05:24 PM Revision 514: input.Makefile: Moved verifications into one set of verify.* files
- 05:18 PM Revision 513: input.Makefile: All tests are run in the same transaction, with output to test.out
- 05:04 PM Revision 512: bin/map: Added support for processing multiple map_paths at once in the same transaction
- 04:50 PM Revision 511: SALVIAS organisms mapping: Removed redundant PlotCode mapping because the association to plotevent is done with PlotID
- 04:49 PM Revision 510: bin/map: Started adding support for multiple map_paths, all run in the same db transaction
- 04:35 PM Revision 509: Filled in SALVIAS stems mappings. make test2's stems test fails because we don't wrap the import transaction around all the tests so that organisms are already in the database when stems are imported.
- 04:28 PM Revision 508: mappings/Makefile: Corrected stemobservation to stem
- 03:46 PM Revision 507: Use baseDistance/complexUserDefined/method to distinguish between different dbhs in VegX
- 03:41 PM Revision 506: VegX mappings: Use baseDistance/value instead of baseDistance so we can later use complexUserDefined to distinguish between different types of dbh
- 03:28 PM Task #335 (Rejected): Look into using Sybase Powerbuilder or IBM Enterprise Vision to map data
- The TACC (Texas Advanced Computing Center) people might have individual licenses they could let us use
- 03:07 PM Revision 505: VegX-VegBIEN.organisms.csv: Added initial stems mappings
- 03:06 PM Revision 504: mappings/Makefile: Updated VegX-VegBIEN.stems.csv input/output roots
- 03:05 PM Revision 503: ch_root: Deal with roots followed by something other than \b
- 02:41 PM Revision 502: input.Makefile: Remove *.out in new verify and test subdirs
- 02:38 PM Revision 501: Moved inputs files into subdirs for maps, verify, and test operations
- 02:00 PM Revision 500: vegbien.sql: Removed taxondetermination.collector_id because that's now in specimen
- 01:58 PM Revision 499: VegBIEN: Map to specimen.collector_id instead of taxondetermination
- 01:27 PM Revision 498: vegbien.sql: Added specimen.collector_id column
- 01:20 PM Revision 497: vegbien.sql: Changed taxondetermination.stem_id to individualplant_id
- 01:16 PM Revision 496: vegbien.sql: Removed taxondetermination columns that have been moved to specimen
- 01:14 PM Revision 495: VegBIEN: Merged specimen.collectionnumber and catalognumber_dwc fields. Removed NOT NULL constraints on several specimen columns because not all specimens have all identifiers. Mapped to new specimen fields.
- 12:59 PM Revision 494: VegX-VegBIEN mapping: Map collectiondate to specimen table
- 12:55 PM Revision 493: vegbien.sql: Added specimen.collectionnumber and collectiondate fields
- 12:50 PM Revision 492: Updated README.TXT to reflect new make targets
01/12/2012
- 04:40 PM Task #334 (Resolved): which DarwinCore attributes do we want to include in the specimen table?
- 04:37 PM Task #314: Import CTFS data
- e-mails on 2012-1-12:
Steve Dolins:
We used the identical aggregate data file used for BIEN I... I think the da... - 04:30 PM Task #333 (Resolved): add traits table to VegBIEN
- 04:25 PM Task #318: decide whether to switch to VegX 1.5.2
- Brad: Yes! Latest version please, 1.5.2. It's about to get a lot less stable in any case, as we make more changes.
- 04:10 PM Task #318 (Resolved): decide whether to switch to VegX 1.5.2
- We have been mapping to VegX 1.5.1 (the latest stable version) but a release candidate VegX 1.5.2 is also available. ...
- 04:24 PM Task #332 (Rejected): switch to VegX 1.5.3
- 04:22 PM Task #331 (Resolved): use VegX's individualOrganismObservation/relatedItem to model stems
- 04:22 PM Task #330 (Rejected): DwC extension to VegX
- 04:22 PM Task #329 (New): make website user-protected
- 04:21 PM Task #328 (New): get raw data rather than flat file exports from RAINFOR
- 04:21 PM Task #327 (Resolved): look into Clio
- 04:21 PM Task #326 (Rejected): generic MOU template to request data
- 04:21 PM Task #325 (Resolved): map CTFS data
- 04:21 PM Task #324 (Resolved): NYBG validations
- 04:20 PM Task #323 (Resolved): map NYBG data
- 04:20 PM Task #322 (Resolved): get access to raw CTFS data
- 04:20 PM Task #321 (Resolved): Convert user-defined VegBIEN fields to first-class fields
- 04:20 PM Task #320 (Rejected): Convert user-defined VegX fields to first-class fields
- 04:20 PM Task #319 (Resolved): Update statistics/lists of user-defined fields in use in VegX and VegBIEN
- 01:25 PM Task #302: Make changes to VegBIEN schema
- I have applied most of Bob's changes to the VegBIEN schema, as well as many of Mark's suggestions. I posted a summary...
- 01:18 PM Revision 491: vegbien.sql: Added sizeclass columns
- 01:01 PM Revision 490: VegBIEN: Removed stem* prefix from stem measurements because it's redundant and because these same field names were reused in non-stem tables
- 12:51 PM Revision 489: VegBIEN: Renamed stemobservation to stem
- 12:49 PM Revision 488: vegbien.sql: Added comments to renamed tables specifying VegBank equivalent
- 12:45 PM Revision 487: vegbien.sql: Added comments to new tables
- 12:25 PM Revision 486: vegbien.sql: Removed r# prefix from foreign key constraints
- 12:21 PM Revision 485: vegbien.sql: Added plot.namedplace_id for theplots which are identified by a namedplace rather than lat/long or name
- 12:18 PM Revision 484: vegbien.sql: Made plot.authorcode optional as not all plots have names
- 12:16 PM Task #317 (Rejected): Direct mapping from VegX to VegBIEN
- When complete, this will be a complete, generic mapping of all elements in VegX to all corresponding elements in VegB...
- 12:15 PM Revision 483: vegbien.sql: Added _dwc to specimen columns that are from DwC
- 12:12 PM Revision 482: vegbien.sql: Added UNIQUE constraint for specimen table. Made catalognumber required so that all specimens can be uniquely identified.
- 12:09 PM Revision 481: vegbien.sql: Made specimen.individualplant_id optional because specimens are not required to have a corresponding plot event (which all individualplants must have)
- 12:05 PM Revision 480: VegBIEN-VegBank.csv: Updated to reflect renames
- 12:02 PM Revision 479: vegbien.sql: Renamed specimen.code to collectioncode to match DwC term
- 11:58 AM Revision 478: vegbien.sql: Added sizeclass table
- 11:51 AM Revision 477: vegbien.sql: Renamed collection to specimen to reflect that it's for a single collection specimen
- 11:45 AM Revision 476: VegBIEN: Renamed collectiveobservation to aggregateoccurrence to reflect VegX usage of aggregate and the new taxonoccurrence table
01/11/2012
- 02:47 PM Revision 475: VegBIEN: Renamed stemobservation.stemcode to authorcode
- 02:43 PM Revision 474: vegbien.sql: Renamed individualplant.code to authorcode
- 02:41 PM Revision 473: sql.py: Reordered get() args to match put()
- 02:40 PM Revision 472: db_xml.py: Use new sql.put() which only eliminates duplicates based on database UNIQUE constraints
- 02:38 PM Revision 471: sql.py: Added new function put() and changed get() to use it
- 02:32 PM Revision 470: sql.py: Fixed raising of NullValueException to pass array of columns rather than single column
- 02:25 PM Revision 469: vegbien.sql: Added UNIQUE constraints to tables used in the mapping so that duplicates are eliminated by the database
- 02:23 PM Revision 468: sql.py: ExceptionWithColumn now stores multiple columns so that they can all be used in DuplicateKeyExceptions
- 02:22 PM Revision 467: util.py: Fixed dict_subset() to work correctly when a key does not exist
- 02:13 PM Revision 466: util.py: Added dict_subset()
- 02:07 PM Revision 465: sql.py: Use constraint_cols() to determine DuplicateKeyException column in try_insert()
- 01:50 PM Revision 464: sql.py: Added constraint_cols() to get columns of a constraint for use in determining the columns for a DuplicateKeyException
- 11:59 AM Revision 463: db_xml.py: Map empty fields to None instead of ignoring them, to ensure that an existing row with a value in that field is not used in the duplicate elimination
- 11:56 AM Revision 462: bin/map: Use db_xml.put()'s new on_error handler
- 11:53 AM Revision 461: db_xml.py: put(): Added on_error handler. Reordered arguments according to how often they are likely to be used.
- 11:31 AM Revision 460: bin/map: Print # input rows processed
- 11:08 AM Task #314: Import CTFS data
- e-mail from Brad Boyle on 2012-1-10:
Interesting observation. I must admit, I haven't inspected Shash's VegX yet, ...
01/10/2012
- 05:32 PM Revision 459: VegX-VegBIEN mapping: Merge different sources of plot elevation
- 05:32 PM Revision 458: vegbien.sql: Added individualplant.code field
- 05:10 PM Revision 457: xml_func.py: Expand XML funcs in XML func args
- 05:06 PM Revision 456: xml_dom.py: In NodeTextEntryIter, return non-text children as-is instead of ignoring them so that XML functions can use them
- 04:46 PM Revision 455: xml_dom.py: Improved readability by separating into sections
- 04:41 PM Revision 454: db_xml.py: Ignore only fields that are actually empty (rather than just not having element children), as a field containing anything other than a comment should be flagged as invalid input
- 04:41 PM Revision 453: xml_dom.py: Changed is_empty() to consider nodes with only comments to be empty
- 04:23 PM Revision 452: bin/map: Add input row to exceptions
- 04:22 PM Revision 451: db_xml.py: Ignore empty fields. Fixes bug where a field that's not text but has no Element children will be treated as a table, and an error is triggered because the corresponding table does not exist in the database.
- 04:19 PM Revision 450: xml_dom.py: Added has_elems() to check if node has Element children
- 03:56 PM Revision 449: bin/map: Use xml_func.process()'s new on_error error handler. Add row to database errors and xml_func.SyntaxExceptions.
- 03:54 PM Revision 448: xml_func.py: Changed formatting of comment that node with syntax errors is replaced with so that it starts on its own line
- 03:50 PM Revision 447: xml_func.py: Use on_error exception handler to allow caller to print exceptions instead of raising them
- 03:41 PM Revision 446: db_xml.py: Add node to any database exceptions generated in put()
- 03:09 PM Revision 445: exc.py: Added raise_() wrapper for raise statement
- 02:45 PM Revision 444: Ignore .~* (lock files generated by OpenOffice.org) in inputs
- 02:44 PM Revision 443: inputs/SALVIAS: Added stems map
- 02:43 PM Revision 442: bin/map: Handle empty rows for db output
- 02:34 PM Revision 441: mappings: Use stemParent instead of stemOrganism to identify stems so that it's clear that the stemParent is an individualOrganismObservation, not an individualOrganism
- 02:25 PM Revision 440: input.Makefile: Include VegX maps not in tablesSort
- 02:20 PM Task #314: Import CTFS data
- e-mail from Mark Schildhauer on 2012-1-10:
I think tt would be best (and certainly most scaleable) if the elements... - 01:20 PM Task #314: Import CTFS data
- Looking at Shash's VegX file, I noticed that she uses different VegX elements than we do for SALVIAS and NYBG data. F...
- 02:13 PM Revision 439: mappings/Makefile: Generate initial VegX-VegBIEN.stems.csv
- 01:36 PM Revision 438: mappings/Makefile: Cleaned up in preparation for adding rule to generate VegX-VegBIEN.stems.csv
- 12:51 PM Revision 437: bin/map: Continue with database insertion even if a row had XML func syntax errors, because these errors have been fixed by removing the offending function node
- 12:49 PM Revision 436: xml_func.py: Remove function nodes with syntax errors to prevent them from causing DatabaseErrors later
- 12:48 PM Revision 435: xml_dom.py: Added remove() function to remove a node
- 12:39 PM Revision 434: bin/map: Track Exceptions with ExTracker
- 12:38 PM Revision 433: exc.py: Added ExTracker to track printed (suppressed) Exceptions and exit with nonzero status if any were encountered
- 12:12 PM Revision 432: bin/map: Parse each output XPath at the beginning so XPath syntax errors are detected right away
- 12:03 PM Revision 431: vegbien.sql: Reran renames to capture cases where a name is in the middle of an identifier
- 11:47 AM Revision 430: vegbien.sql: Use character varying(255) instead of text or character varying in new tables and columns
- 11:35 AM Revision 429: input.Makefile: Print Postgres NULL as NULL instead of empty string to match MySQL output
- 11:26 AM Revision 428: VegX-VegBIEN mapping: Fixed syntax error in mapping
01/09/2012
- 05:30 PM Revision 427: vegbien.sql: Added trait table
- 05:22 PM Revision 426: vegbien.sql: Added columns to stemobservation
- 05:13 PM Revision 425: vegbien.sql: Renamed collectiveobservation.stratum_id to taxonbin_id
- 04:51 PM Revision 424: vegbien.sql: Added some DarwinCore attributes to collection table
- 04:44 PM Task #302: Make changes to VegBIEN schema
- Added new tables and moved/updated columns
- 04:44 PM Task #302: Make changes to VegBIEN schema
- I asked Bob a few questions about the new Collection table in the VegBIEN modifications:
* Is the Collection table m... - 04:21 PM Revision 423: Makefiles: Added remake target to run clean and all
- 04:04 PM Revision 422: vegbien.sql: Added taxondetermination.collection_id column
- 03:56 PM Revision 421: VegBIEN: Made taxondetermination.plantname_id required
- 02:59 PM Revision 420: vegbien.sql: Made taxondetermination.plantconcept_id optional
- 02:49 PM Revision 419: VegBIEN: Renamed author*code to authorcode
- 02:47 PM Revision 418: Makefiles: Added empty rules for targets without a rule so that these rules wouldn't trigger the recursive subdir invocation for the default target
- 02:33 PM Revision 417: VegX-VegBIEN mapping: Convereted all database names to lowercase to correspond to their case in the database
- 02:04 PM Revision 416: Makefiles: Added empty Makefile target so that Makefiles are not included in the targets recursively run on subdirs
- 02:01 PM Revision 415: input.Makefile: Dependencies in other directories are included as prerequisistes. Use tablesSort variable instead of sort order prefix to order tables.
- 01:35 PM Revision 414: Makefile: Added default target that recursed into subdirs with Makefiles. Used default target for all and clean.
- 01:34 PM Revision 413: Regenerated inputs/SALVIAS/map.VegBIEN.0.plots.csv
- 01:32 PM Revision 412: Removed scripts symlink because it was causing Makefile recursion issues
- 01:28 PM Revision 411: Makefile: Take advantage of new inputs/Makefile
- 01:24 PM Revision 410: Added inputs/Makefile to run input targets recursively on each input
- 01:08 PM Revision 409: Makefile: Run inputs-* targets for all and clean
- 01:00 PM Revision 408: input.Makefile: all doesn't run import. import runs verify.
- 12:41 PM Revision 407: vegbien.sql: Set all FOREIGN KEY constraints to ON UPDATE CASCADE ON DELETE CASCADE
- 12:35 PM Revision 406: SALVIAS mappings: use PlotID as authorObsCode to link plot observations and organisms correctly for organisms without a PlotCode
- 12:32 PM Revision 405: SALVIAS mappings: use PlotID as authorObsCode to link plot observations and organisms correctly for organisms without a PlotCode
- 11:49 AM Revision 404: input.Makefile: Fixed bug in verify's diff rule
- 11:40 AM Task #314: Import CTFS data
- I uploaded Shash's VegX file to vegbiendev at @/home/bien/svn/inputs/CTFS/src.VegX.xml@ .
- 11:34 AM Revision 403: Added CTFS input
01/06/2012
- 05:01 PM Revision 402: Makefile: Generate error when trying to run subMake on a directory with no Makefile
- 04:37 PM Revision 401: input.Makefile: Tests use bin/map's new new test mode
- 04:37 PM Revision 400: bin/map: Added test mode that empties the database before doing anything
- 04:37 PM Revision 399: sql.py: Added functions to truncate tables, list all tables, and empty a database
- 04:35 PM Revision 398: Accepted new SALVIAS test outputs
- 03:38 PM Revision 397: Makefile: Print Finished message after running test2 (inputs tests)
- 03:38 PM Revision 396: Accepted test outputs for SALVIAS
- 03:37 PM Revision 395: input.Makefile: Added ability to accept a test output to use as the new reference output
- 03:01 PM Revision 394: vegbien db: Renamed all occurrences of interpretation to determination
- 02:56 PM Revision 393: Makefile: Tell make that subMake rules run make recursively
- 02:51 PM Revision 392: vegbien.sql: Updated collection table for recent refactorings
- 02:42 PM Revision 391: Made more of Bob Peet's changes to the vegbien db and updated VegX-VegBIEN mapping
- 02:32 PM Revision 390: Makefile: Fixed pattern for recursive invocation of subdirectory make
- 12:54 PM Revision 389: Moved mappings/schemas to schemas
- 12:50 PM Revision 388: Makefile: Changed all to make schemas and mappings instead of running install
- 12:34 PM Revision 387: Moved mappings/schemas to schemas
- 12:25 PM Revision 386: Makefile: Added inputs import target
- 12:24 PM Revision 385: input.Makefile: Fixed syntax error with use of import_cmd variable
- 12:19 PM Revision 384: vegbien.sql: Added collection table
- 12:18 PM Revision 383: Makefiles: Changed _not_file to _always because _not_file is sometimes used to force a file to always be remade
01/05/2012
- 04:53 PM Task #311 (Resolved): Decouple VegBIEN from VegBank and map directly from VegX to VegBIEN
- Decoupled VegBIEN from VegBank and removed mappings to VegBank
- 03:07 PM Task #311 (Resolved): Decouple VegBIEN from VegBank and map directly from VegX to VegBIEN
- 04:48 PM Revision 382: Decoupled VegBIEN from VegBank and removed mappings to VegBank
- 04:30 PM Revision 381: Regenerate vegbien.sql by dumping the vegbien db
- 04:06 PM Revision 380: Regenerate vegbien.sql by dumping the vegbien db
- 04:05 PM Revision 379: psql_vegbien: Changed to use postgres_vegbien
- 04:03 PM Revision 378: Added postgres_vegbien to run PostgreSQL command on the vegbien db
- 03:26 PM Revision 377: SALVIAS verification: Added plot methods
- 03:09 PM Task #315 (New): Import TurboVeg data
- 03:09 PM Task #314 (Resolved): Import CTFS data
- * coordinate with Shash
* CTFS has a lot of stems data - 03:08 PM Task #313 (Resolved): Import full NYBG data
- 03:08 PM Task #312 (Resolved): Finish importing SALVIAS data
- # -Import stems data-
# -Fix data format issues-: _Not sure what these were, probably long fixed_
# -Map invalid da... - 01:19 PM Revision 376: SALVIAS verification: Added # plots
- 01:06 PM Revision 375: Makefile: Added targets to run input targets on all inputs
- 12:54 PM Revision 374: SALVIAS verification: Added projects list to examine why project counts differ
- 12:41 PM Revision 373: SALVIAS verification: Made statement-label SELECT statements use the same syntax for MySQL and PostgreSQL so they are easy to copy and paste between the input and output queries
- 11:26 AM Task #290: benchmark tests for database loading
- You can view the validation results by running the following on vegbiendev:
@make verify --directory=/home/bien/svn/... - 11:25 AM Task #305: Complete full-dataset validations for NYBG & SALVIAS
- You can view the validation results by running the following on vegbiendev:
@make verify --directory=/home/bien/svn/... - 11:25 AM Task #302: Make changes to VegBIEN schema
- The vegbien db now contains some of the modifications Bob Peet made to the VegBank schema for VegBIEN, so some table ...
- 11:24 AM Task #304: Complete full dataset imports to VegBIEN via VegX of NYBG and SALVIAS
- I loaded the SALVIAS data (plots and organisms) into VegBIEN on vegbiendev. To browse the data, follow the [[phpPgAdm...
01/04/2012
- 05:00 PM Task #305: Complete full-dataset validations for NYBG & SALVIAS
- Created mechanism to write verification queries and compare to desired output
- 04:57 PM Task #304: Complete full dataset imports to VegBIEN via VegX of NYBG and SALVIAS
- Core SALVIAS data (plots and organisms) are on vegbiendev in VegBIEN.
Takes overnight to reinsert organisms data whe... - 04:54 PM Revision 372: input.Makefile: Don't remove logs in clean
- 04:49 PM Task #303: Mapping from VegBIEN to original VegBank
- Search-and-replace operations that transform VegBank to VegBIEN can in many cases be reversed to transform VegBIEN to...
- 04:48 PM Task #302: Make changes to VegBIEN schema
- All functions that used to output to VegBank now output to VegBIEN db.
The VegBIEN schema is automatically generated... - 04:45 PM Task #296: Direct mapping from native salvias_plots MySQL database to VegBIEN
- We will also need to import the projects, stems, and lookup_MethodCode tables in order for all the verification queri...
- 04:43 PM Task #290: benchmark tests for database loading
- Created mechanism to write verification queries and compare to desired output
- 04:32 PM Revision 371: input.Makefile: Added support for multi-column verification queries. inputs/SALVIAS: Added a multi-column verification query.
- 03:47 PM Revision 370: input.Makefile: Remove all *.out in clean
- 02:59 PM Revision 369: inputs: Don't keep *.out under version control
- 02:35 PM Revision 368: input.Makefile: Added rules to verify inserted data and run tests
- 11:38 AM Revision 367: Regenerated inputs/SALVIAS/map.VegBIEN.1.organisms.csv
- 11:32 AM Revision 366: exc.py: Print ! before error messages so they can be found and counted more easily in log files
- 11:13 AM Revision 365: input.Makefile: Only output to log if importing all rows
- 10:46 AM Revision 364: Moved vegbien CREATE DATABASE command to Makefile and run vegbien.sql as bien user
- 10:27 AM Revision 363: Renamed vegbien tables according to Bob Peet's new VegBank-VegBIEN mapping
- 10:16 AM Revision 362: input.Makefile: Don't print commands that use set -x to be echoed
- 10:14 AM Revision 361: input.Makefile: Use system's time utility instead of shell's time command
- 10:09 AM Revision 360: sql.py: Include sql.DbException in DatabaseErrors list
01/03/2012
- 04:28 PM Revision 359: Generate VegX-VegBIEN.organisms.csv from VegBank-VegBIEN.map.csv, which is generated from VegBank-VegBIEN.map_only.csv
- 04:21 PM Revision 358: repl: Process whole file as one string so that regexps can match EOF for appending
- 04:21 PM Revision 357: mappings/Makefile: Clean *-VegBIEN.*.csv maps
- 04:08 PM Revision 356: Generate VegBank-VegBIEN.schema.csv from VegBank-VegBIEN.schema_only.csv and VegBank-VegBIEN.csv
- 04:06 PM Revision 355: Added cat_csv to concatenate two spreadsheets
- 03:34 PM Revision 354: Removed no longer needed inputs/SALVIAS/db.sh
- 03:24 PM Revision 353: Add sort order prefixes to inputs/SALVIAS maps
- 02:49 PM Revision 352: VegX-VegBank organisms mapping: Added collectionDate mapping
- 02:36 PM Revision 351: VegX-VegBank organisms mapping: Added collectionDate mapping
- 02:36 PM Revision 350: mappings/Makefile: Sort VegX-VegBank organisms mapping automatically when anything that depends on it is remade
- 02:34 PM Revision 349: Added in_place to run a stream command on a file
- 02:14 PM Revision 348: join_sort: Use /bin/sh instead of /bin/bash
- 01:44 PM Revision 347: NYBG-VegX mapping: Removed no longer needed global stem count value
- 01:42 PM Revision 346: Renamed test/accept_output to accept
- 01:40 PM Revision 345: VegX-VegBank.organisms.csv: Made stemCount default to 1
- 01:17 PM Revision 344: bin/map: Print DatabaseErrors without stack traces
- 01:13 PM Revision 343: exc.py: Added print_ex()
- 01:13 PM Revision 342: exc.py: Added print_ex()
- 01:13 PM Revision 341: util.py: Added type_name()
- 01:12 PM Revision 340: strings.py: Added ensure_newl()
- 11:42 AM Revision 339: input.Makefile: Save map output in log file
- 11:27 AM Revision 338: bin/map: Print usage message when run with no arguments
- 10:10 AM Revision 337: Makefile: Removed unnecessary commenting out of "allow from 127.0.0.0..." lines in /etc/phppgadmin/apache.conf
12/29/2011
- 05:35 PM Revision 336: Makefile: Use --loose-exit-status for ed to avoid failing on no match. In editPhppgadminApacheConf, use ,s/// instead of /.../\ns/// to match lines regardless of position.
- 05:17 PM Revision 335: Makefile: Use tee instead of ed in sudoAppend. Don't change apache port to 8080 if apache already running on 80.
- 04:54 PM Revision 334: Makefile: Added phppgadmin apache config
- 01:16 PM Revision 333: bin/map: Only treat DatabaseErrors as warnings, not all Exceptions
- 01:04 PM Revision 332: xml_func.py: Print newline before func XML because XML is now pretty-printed with indents
- 12:59 PM Revision 331: xml_dom.py: Node.__str__() uses toprettyxml() instead of toxml()
- 12:58 PM Revision 330: util.py: Changed rename_key() not to suppress KeyError, since some callers might want to assert that a key exists
- 12:44 PM Revision 329: Makefile: Added phpPgAdmin dependency
- 12:42 PM Revision 328: xml_func.py: _name func doesn't require all name parts to be set. _namePart func throws SyntaxException for invalid name parts.
12/28/2011
- 02:03 PM Revision 327: bin/map: Fixed syntax error where db variable had not been renamed to out_db
- 01:58 PM Revision 326: xml_func.py: Generate SyntaxExceptions on missing XML func args
- 01:46 PM Revision 325: VegX-VegBank mapping: Added default soilObs.soilHorizon value
- 01:33 PM Revision 324: xml_func.py: Changed _date func to try parsing date field, rather than year field, as a decimal. Changed SALVIAS mappings to use new _date func syntax. Fixes SALVIAS plots data syntax error for decimal years.
- 01:10 PM Revision 323: Removed no longer needed VegBank accepted test outputs
- 01:09 PM Revision 322: test/map: Abort tester on failed diff
- 01:09 PM Revision 321: Added accepted test output SALVIAS_db.organisms.VegBIEN.xml
- 01:04 PM Revision 320: SALVIAS organisms mapping: Use _alt to merge tag1 and tag2. Fixes failing diff due to duplicate stemCodes.
- 12:49 PM Revision 319: test/map: Added tests to insert VegBIEN XML into VegBIEN db
- 12:47 PM Revision 318: bin/map: Fixed non-map input to set out_label
- 12:12 PM Revision 317: bin/map: Fixed row processing when no map is used
- 12:06 PM Revision 316: bin/map: Refactored to process each row separately when out_is_db
- 12:04 PM Revision 315: xml_dom.py: Added Node pop() and clear() methods
- 11:15 AM Revision 314: bin/map: Moved process_rows() exception handling around entire row, rather than each value, so that a broken value will not be ignored only to cause errors when outputing the row
- 11:12 AM Revision 313: bin/map: Moved process_rows() exception handling around entire row, rather than each value, so that a broken value will not be ignored only to cause errors when outputing the row
- 11:05 AM Revision 312: db_xml.py: Removed no longer needed xml2db()
- 11:04 AM Revision 311: bin/map: out_is_db uses use_row instead of xml2db()
- 10:53 AM Revision 310: bin/map: Added commit and exception-handling code to out_is_db use_row()
- 10:42 AM Revision 309: bin/map: Merged get_xml() into process_xml(). Renamed process*() functions to distinguish them better.
- 10:33 AM Revision 308: bin/map: Create separate document for printing XML opening and closing
- 10:28 AM Revision 307: bin/map: Refactored to print document opening, contents, and closing in separate steps so they can be separated later
- 10:27 AM Revision 306: xml_dom.py: Added support for integer indents
- 10:19 AM Revision 305: xml_dom.py: Added NodeIter() and Node.__iter__()
- 10:05 AM Revision 304: Use new xml_dom.prettyxml_config instead of Node.writeprettyxml because it can be passed as keyword args to any writexml function
- 10:00 AM Revision 303: xml_dom.py: Removed no longer needed xml_dom.writexml()
- 10:00 AM Revision 302: bin/map: Use Node.writeprettyxml() instead of xml_dom.writexml()
- 09:57 AM Revision 301: xml_dom.py: Added minidom.Document write_opening() and write_closing() methods. Changed writexml(writer, node) to use new Node.writeprettyxml() method.
- 09:28 AM Revision 300: Renamed ex.py to exc.py
- 09:24 AM Revision 299: Import xml.dom.minidom as minidom
- 09:10 AM Revision 298: xml_dom.py: Refactored xml.dom.minidom.Element.writexml() to use new write_opening() and write_closing() methods. Use new functions is_elem() and is_text_node() to determine node type. Corrected definition of a text element to be an element that contains a single text node.
- 08:37 AM Revision 297: bin/map: Factored out input processing code
12/23/2011
- 04:03 PM Revision 296: Commit after inserting each row
- 03:59 PM Revision 295: Commit after inserting each row
- 03:54 PM Revision 294: Use traceback.print_exc() to print Exceptions for each row instead of exiting
- 03:26 PM Revision 293: env_password: Allow user to change saved password
- 03:14 PM Revision 292: env_password: Don't print instructions for storing password for the session if called *to* store password for the session
- 03:00 PM Revision 291: env_password: Print absolute path to env_password in case invoking script changed the working directory
- 02:57 PM Revision 290: env_password: Print instructions for storing password for the session
- 02:32 PM Revision 289: Updated SALVIAS plots mappings and corresponding VegX-VegBank mappings to parse date_start and date_finish
- 02:11 PM Revision 288: xml_func.py: Print XML of XML functions with SyntaxExceptions
- 02:08 PM Revision 287: xml_func.py: Added support for parsing dates
- 01:56 PM Revision 286: ex.py: repl_msg(): take replacements as keyword args
- 01:44 PM Revision 285: fix_permissions: Also change user to bien
- 01:31 PM Revision 284: Makefile: Added python-dateutil dependency
- 12:57 PM Revision 283: input.Makefile: Renamed from_db-<map_file> target to import-<type> so it can be more easily run as a command-line target rather than just a prerequisite
- 12:40 PM Revision 282: Added inputs/SALVIAS organisms map
12/22/2011
- 08:30 PM Revision 281: map: Print xml_func.SyntaxExceptions without stack traces by using SystemExit
- 08:22 PM Revision 280: xml_func.py: Add function name to SyntaxException message
- 08:22 PM Revision 279: ex.py: Added repl_msg() to format a message with the % operator
- 07:48 PM Revision 278: xml_func.py: Return string->number conversion errors as xml_func.SyntaxExceptions
- 07:29 PM Revision 277: psql_vegbien: Fixed comment to vegbien instead of vegbank
- 07:11 PM Revision 276: psql_vegbien: Use new location of bien_password
- 07:05 PM Revision 275: Makefile: Fixed paths to mappings dir for new scripts dir location
- 07:02 PM Revision 274: Renamed util to bin
- 06:59 PM Revision 273: Moved inputs_Makefile to inputs/input.Makefile
- 06:55 PM Revision 272: Moved bien_password to new config dir
- 06:52 PM Revision 271: Moved sample inputs to test dir
- 06:42 PM Revision 270: Added symlink from scripts to new scripts destination
- 06:40 PM Revision 269: Removed now-empty scripts dir
- 06:39 PM Revision 268: Moved everything in scripts to root. inputs_Makefile: Don't run "all" when installing.
- 06:24 PM Revision 267: Renamed bien_map to map
- 06:22 PM Revision 266: Moved map to util
- 06:14 PM Revision 265: fix_permissions: Don't chmod symlinks
- 06:00 PM Revision 264: inputs_Makefile: Auto-generate map to VegBIEN and import data into vegbien from input DB
- 05:59 PM Revision 263: inputs/SALVIAS: maps to VegX and VegBIEN
- 05:58 PM Revision 262: bien_map: Runs map with BIEN defaults
- 05:56 PM Revision 261: join_sort: Sorts a join on the output col
- 03:20 PM Revision 260: inputs_Makefile: Require dbEngine var instead of defaulting to MySQL
- 02:22 PM Revision 259: Moved inputs into svn
- 01:46 PM Revision 258: Moved pre-BIEN 3 files into _archive folder
- 01:46 PM Revision 257: test/map: Use db.sh syntax in *.sh tests
- 01:45 PM Revision 256: inputs_Makefile: Generate db.sh with DB access info
- 12:59 PM Revision 255: inputs_Makefile: Changed GRANT ALL to GRANT SELECT. Added REVOKE ALL. Added $(db).sql as prerequisite of install in case it needs to be auto-generated.
- 12:41 PM Task #302: Make changes to VegBIEN schema
- Bob Peet's changes are at [[VegBIEN from VegBank]]
E-mail from Bob Peet on 2011-12-22:
I promised to summarize ... - 12:31 PM Task #310 (Resolved): automated build process
- I set up an automated build process for all the BIEN scripts and dependencies. It runs on both vegbiendev and nimoy. ...
- 12:31 PM Task #310 (Resolved): automated build process
- 12:30 PM Revision 254: scripts/Makefile: Added action for postgres-Darwin target
- 12:11 PM Revision 253: scripts/Makefile: Added postgresql to postgres-Linux apt-get packages
12/21/2011
- 08:23 PM Revision 252: test/map: Changed to work on both nimoy and vegbiendev by selecting the appropriate MySQL user and password
- 08:22 PM Revision 251: env_password: Added optional message arg
- 07:47 PM Revision 250: Added uninstallation of inputs to Makefiles
- 07:15 PM Revision 249: scripts/Makefile: Create bien user w/o prompting for password. Fixed syntax error.
- 07:06 PM Revision 248: scripts/Makefile: Fixed syntax error
- 07:03 PM Revision 247: Inputs now include inputs_Makefile to get mysql command, etc.
- 06:30 PM Revision 246: scripts/Makefile: Use bien MySQL user for installing inputs
- 06:26 PM Revision 245: scripts/Makefile: Use root MySQL user for creating bien user
- 05:53 PM Revision 244: scripts/Makefile: Don't use root as MySQL admin user. Removed no-longer-needed sub-makes for setting DB login vars.
- 05:10 PM Revision 243: Changed vegbien_dest and users of it to use separate bien_password file
- 04:55 PM Revision 242: fix_permissions: Extend all user permissions to group
- 04:26 PM Revision 241: scripts/Makefile: Added inputs
- 03:24 PM Revision 240: mappings/Makefile: Cleaned up
- 03:02 PM Revision 239: fix_permissions: Don't add group write perms to read-only files
- 01:54 PM Revision 238: scripts/Makefile: Made read command syntax compatible with /bin/sh
- 01:37 PM Revision 237: scripts/Makefile: Ignore errors about missing packages
- 01:30 PM Revision 236: scripts/Makefile: Fixed escape character for /bin/sh
- 01:25 PM Revision 235: scripts/Makefile: Added dependency installation. Makefiles: Use _not_file instead of FORCE for clarity. Use $(os) var
12/20/2011
- 07:52 PM Revision 234: fix_permissions: Configured output verbosity
- 07:44 PM Revision 233: Added fix_permissions to set correct permissions on shared bien files
- 07:31 PM Revision 232: Removed VegBank scripts which are no longer needed
- 07:30 PM Revision 231: Removed VegBank scripts which are no longer needed
- 06:55 PM Revision 230: Renamed vegbien_dest.sh to vegbien_dest to reflect that it is also includable by Makefiles
- 06:52 PM Revision 229: DB user creation: Clarified instructions
- 06:51 PM Revision 228: scripts/Makefile includes vegbien_dest.sh directly
- 06:28 PM Revision 227: Moved db user creation to scripts/Makefile. Removed now-unneeded admin scripts.
- 06:12 PM Revision 226: Removed VegBank scripts which are no longer needed
- 06:11 PM Revision 225: scripts/Makefile: Added empty_db target which uses vegbien_empty.sql
- 05:55 PM Revision 224: mappings/Makefile: Fixed bug where var containing prerequisistes needed to be defined before used. Added support for different sed flags to use extended regular expressions.
- 05:42 PM Revision 223: Added auto-generated vegbien_empty.sql to empty the vegbien db
- 04:55 PM Revision 222: Test output to VegBIEN instead of VegBank
- 04:54 PM Revision 221: VegX-VegBIEN.organisms.csv: sort output of repl
- 04:46 PM Revision 220: review: Added nullglob
- 04:44 PM Revision 219: review: Don't process replacements spreadsheets
- 04:44 PM Revision 218: Moved schema replacements from VegBank-VegBIEN.csv to VegBank-VegBIEN.schema.csv
- 04:32 PM Revision 217: VegBank-VegBIEN.csv: Support PostgreSQL before 8.4
- 04:29 PM Revision 216: repl: Added support for blank lines. Only add whole word regexp code to inputs w/o *any* regexp metachars.
- 04:13 PM Revision 215: Create vegbien db from mappings/schemas/vegbien.sql
- 04:12 PM Revision 214: VegBank-VegBIEN.csv: Added replacements for SQL create script conversion
- 03:56 PM Revision 213: Generate vegbien db create SQL from vegbank.sql using repl
- 03:55 PM Revision 212: Generate vegbien db create SQL from vegbank.sql using repl
- 03:54 PM Revision 211: repl: Fixed bug in reading arguments
- 03:38 PM Revision 210: repl: Don't add whole-word regexp for inputs that already have regexp metachars
- 03:26 PM Revision 209: Removed mappings/VegBank-VegBIEN.organisms.csv because now using replacements spreadsheet
- 03:24 PM Revision 208: mappings: Generate mappings to VegBIEN using VegBank-VegBIEN.cs replacements spreadsheet
- 03:23 PM Revision 207: mappings: Generate mappings to VegBIEN using VegBank-VegBIEN.cs replacements spreadsheet
- 03:23 PM Revision 206: mappings: Generate mappings to VegBIEN using VegBank-VegBIEN.cs replacements spreadsheet
- 02:57 PM Revision 205: Added repl to perform replacements on a spreadsheet or file
- 01:38 PM Revision 204: scripts/Makefile: removed extra test-% target
- 01:38 PM Revision 203: README.TXT: Updated
- 01:36 PM Revision 202: scripts/Makefile: install/uninstall targets
- 01:36 PM Revision 201: bien_user_create: Print instructions in sequence with password prompts
Also available in: Atom