Activity
From 12/12/2011 to 01/10/2012
01/10/2012
- 05:32 PM Revision 459: VegX-VegBIEN mapping: Merge different sources of plot elevation
- 05:32 PM Revision 458: vegbien.sql: Added individualplant.code field
- 05:10 PM Revision 457: xml_func.py: Expand XML funcs in XML func args
- 05:06 PM Revision 456: xml_dom.py: In NodeTextEntryIter, return non-text children as-is instead of ignoring them so that XML functions can use them
- 04:46 PM Revision 455: xml_dom.py: Improved readability by separating into sections
- 04:41 PM Revision 454: db_xml.py: Ignore only fields that are actually empty (rather than just not having element children), as a field containing anything other than a comment should be flagged as invalid input
- 04:41 PM Revision 453: xml_dom.py: Changed is_empty() to consider nodes with only comments to be empty
- 04:23 PM Revision 452: bin/map: Add input row to exceptions
- 04:22 PM Revision 451: db_xml.py: Ignore empty fields. Fixes bug where a field that's not text but has no Element children will be treated as a table, and an error is triggered because the corresponding table does not exist in the database.
- 04:19 PM Revision 450: xml_dom.py: Added has_elems() to check if node has Element children
- 03:56 PM Revision 449: bin/map: Use xml_func.process()'s new on_error error handler. Add row to database errors and xml_func.SyntaxExceptions.
- 03:54 PM Revision 448: xml_func.py: Changed formatting of comment that node with syntax errors is replaced with so that it starts on its own line
- 03:50 PM Revision 447: xml_func.py: Use on_error exception handler to allow caller to print exceptions instead of raising them
- 03:41 PM Revision 446: db_xml.py: Add node to any database exceptions generated in put()
- 03:09 PM Revision 445: exc.py: Added raise_() wrapper for raise statement
- 02:45 PM Revision 444: Ignore .~* (lock files generated by OpenOffice.org) in inputs
- 02:44 PM Revision 443: inputs/SALVIAS: Added stems map
- 02:43 PM Revision 442: bin/map: Handle empty rows for db output
- 02:34 PM Revision 441: mappings: Use stemParent instead of stemOrganism to identify stems so that it's clear that the stemParent is an individualOrganismObservation, not an individualOrganism
- 02:25 PM Revision 440: input.Makefile: Include VegX maps not in tablesSort
- 02:20 PM Task #314: Import CTFS data
- e-mail from Mark Schildhauer on 2012-1-10:
I think tt would be best (and certainly most scaleable) if the elements... - 01:20 PM Task #314: Import CTFS data
- Looking at Shash's VegX file, I noticed that she uses different VegX elements than we do for SALVIAS and NYBG data. F...
- 02:13 PM Revision 439: mappings/Makefile: Generate initial VegX-VegBIEN.stems.csv
- 01:36 PM Revision 438: mappings/Makefile: Cleaned up in preparation for adding rule to generate VegX-VegBIEN.stems.csv
- 12:51 PM Revision 437: bin/map: Continue with database insertion even if a row had XML func syntax errors, because these errors have been fixed by removing the offending function node
- 12:49 PM Revision 436: xml_func.py: Remove function nodes with syntax errors to prevent them from causing DatabaseErrors later
- 12:48 PM Revision 435: xml_dom.py: Added remove() function to remove a node
- 12:39 PM Revision 434: bin/map: Track Exceptions with ExTracker
- 12:38 PM Revision 433: exc.py: Added ExTracker to track printed (suppressed) Exceptions and exit with nonzero status if any were encountered
- 12:12 PM Revision 432: bin/map: Parse each output XPath at the beginning so XPath syntax errors are detected right away
- 12:03 PM Revision 431: vegbien.sql: Reran renames to capture cases where a name is in the middle of an identifier
- 11:47 AM Revision 430: vegbien.sql: Use character varying(255) instead of text or character varying in new tables and columns
- 11:35 AM Revision 429: input.Makefile: Print Postgres NULL as NULL instead of empty string to match MySQL output
- 11:26 AM Revision 428: VegX-VegBIEN mapping: Fixed syntax error in mapping
01/09/2012
- 05:30 PM Revision 427: vegbien.sql: Added trait table
- 05:22 PM Revision 426: vegbien.sql: Added columns to stemobservation
- 05:13 PM Revision 425: vegbien.sql: Renamed collectiveobservation.stratum_id to taxonbin_id
- 04:51 PM Revision 424: vegbien.sql: Added some DarwinCore attributes to collection table
- 04:44 PM Task #302: Make changes to VegBIEN schema
- Added new tables and moved/updated columns
- 04:44 PM Task #302: Make changes to VegBIEN schema
- I asked Bob a few questions about the new Collection table in the VegBIEN modifications:
* Is the Collection table m... - 04:21 PM Revision 423: Makefiles: Added remake target to run clean and all
- 04:04 PM Revision 422: vegbien.sql: Added taxondetermination.collection_id column
- 03:56 PM Revision 421: VegBIEN: Made taxondetermination.plantname_id required
- 02:59 PM Revision 420: vegbien.sql: Made taxondetermination.plantconcept_id optional
- 02:49 PM Revision 419: VegBIEN: Renamed author*code to authorcode
- 02:47 PM Revision 418: Makefiles: Added empty rules for targets without a rule so that these rules wouldn't trigger the recursive subdir invocation for the default target
- 02:33 PM Revision 417: VegX-VegBIEN mapping: Convereted all database names to lowercase to correspond to their case in the database
- 02:04 PM Revision 416: Makefiles: Added empty Makefile target so that Makefiles are not included in the targets recursively run on subdirs
- 02:01 PM Revision 415: input.Makefile: Dependencies in other directories are included as prerequisistes. Use tablesSort variable instead of sort order prefix to order tables.
- 01:35 PM Revision 414: Makefile: Added default target that recursed into subdirs with Makefiles. Used default target for all and clean.
- 01:34 PM Revision 413: Regenerated inputs/SALVIAS/map.VegBIEN.0.plots.csv
- 01:32 PM Revision 412: Removed scripts symlink because it was causing Makefile recursion issues
- 01:28 PM Revision 411: Makefile: Take advantage of new inputs/Makefile
- 01:24 PM Revision 410: Added inputs/Makefile to run input targets recursively on each input
- 01:08 PM Revision 409: Makefile: Run inputs-* targets for all and clean
- 01:00 PM Revision 408: input.Makefile: all doesn't run import. import runs verify.
- 12:41 PM Revision 407: vegbien.sql: Set all FOREIGN KEY constraints to ON UPDATE CASCADE ON DELETE CASCADE
- 12:35 PM Revision 406: SALVIAS mappings: use PlotID as authorObsCode to link plot observations and organisms correctly for organisms without a PlotCode
- 12:32 PM Revision 405: SALVIAS mappings: use PlotID as authorObsCode to link plot observations and organisms correctly for organisms without a PlotCode
- 11:49 AM Revision 404: input.Makefile: Fixed bug in verify's diff rule
- 11:40 AM Task #314: Import CTFS data
- I uploaded Shash's VegX file to vegbiendev at @/home/bien/svn/inputs/CTFS/src.VegX.xml@ .
- 11:34 AM Revision 403: Added CTFS input
01/06/2012
- 05:01 PM Revision 402: Makefile: Generate error when trying to run subMake on a directory with no Makefile
- 04:37 PM Revision 401: input.Makefile: Tests use bin/map's new new test mode
- 04:37 PM Revision 400: bin/map: Added test mode that empties the database before doing anything
- 04:37 PM Revision 399: sql.py: Added functions to truncate tables, list all tables, and empty a database
- 04:35 PM Revision 398: Accepted new SALVIAS test outputs
- 03:38 PM Revision 397: Makefile: Print Finished message after running test2 (inputs tests)
- 03:38 PM Revision 396: Accepted test outputs for SALVIAS
- 03:37 PM Revision 395: input.Makefile: Added ability to accept a test output to use as the new reference output
- 03:01 PM Revision 394: vegbien db: Renamed all occurrences of interpretation to determination
- 02:56 PM Revision 393: Makefile: Tell make that subMake rules run make recursively
- 02:51 PM Revision 392: vegbien.sql: Updated collection table for recent refactorings
- 02:42 PM Revision 391: Made more of Bob Peet's changes to the vegbien db and updated VegX-VegBIEN mapping
- 02:32 PM Revision 390: Makefile: Fixed pattern for recursive invocation of subdirectory make
- 12:54 PM Revision 389: Moved mappings/schemas to schemas
- 12:50 PM Revision 388: Makefile: Changed all to make schemas and mappings instead of running install
- 12:34 PM Revision 387: Moved mappings/schemas to schemas
- 12:25 PM Revision 386: Makefile: Added inputs import target
- 12:24 PM Revision 385: input.Makefile: Fixed syntax error with use of import_cmd variable
- 12:19 PM Revision 384: vegbien.sql: Added collection table
- 12:18 PM Revision 383: Makefiles: Changed _not_file to _always because _not_file is sometimes used to force a file to always be remade
01/05/2012
- 04:53 PM Task #311 (Resolved): Decouple VegBIEN from VegBank and map directly from VegX to VegBIEN
- Decoupled VegBIEN from VegBank and removed mappings to VegBank
- 03:07 PM Task #311 (Resolved): Decouple VegBIEN from VegBank and map directly from VegX to VegBIEN
- 04:48 PM Revision 382: Decoupled VegBIEN from VegBank and removed mappings to VegBank
- 04:30 PM Revision 381: Regenerate vegbien.sql by dumping the vegbien db
- 04:06 PM Revision 380: Regenerate vegbien.sql by dumping the vegbien db
- 04:05 PM Revision 379: psql_vegbien: Changed to use postgres_vegbien
- 04:03 PM Revision 378: Added postgres_vegbien to run PostgreSQL command on the vegbien db
- 03:26 PM Revision 377: SALVIAS verification: Added plot methods
- 03:09 PM Task #315 (New): Import TurboVeg data
- 03:09 PM Task #314 (Resolved): Import CTFS data
- * coordinate with Shash
* CTFS has a lot of stems data - 03:08 PM Task #313 (Resolved): Import full NYBG data
- 03:08 PM Task #312 (Resolved): Finish importing SALVIAS data
- # -Import stems data-
# -Fix data format issues-: _Not sure what these were, probably long fixed_
# -Map invalid da... - 01:19 PM Revision 376: SALVIAS verification: Added # plots
- 01:06 PM Revision 375: Makefile: Added targets to run input targets on all inputs
- 12:54 PM Revision 374: SALVIAS verification: Added projects list to examine why project counts differ
- 12:41 PM Revision 373: SALVIAS verification: Made statement-label SELECT statements use the same syntax for MySQL and PostgreSQL so they are easy to copy and paste between the input and output queries
- 11:26 AM Task #290: benchmark tests for database loading
- You can view the validation results by running the following on vegbiendev:
@make verify --directory=/home/bien/svn/... - 11:25 AM Task #305: Complete full-dataset validations for NYBG & SALVIAS
- You can view the validation results by running the following on vegbiendev:
@make verify --directory=/home/bien/svn/... - 11:25 AM Task #302: Make changes to VegBIEN schema
- The vegbien db now contains some of the modifications Bob Peet made to the VegBank schema for VegBIEN, so some table ...
- 11:24 AM Task #304: Complete full dataset imports to VegBIEN via VegX of NYBG and SALVIAS
- I loaded the SALVIAS data (plots and organisms) into VegBIEN on vegbiendev. To browse the data, follow the [[phpPgAdm...
01/04/2012
- 05:00 PM Task #305: Complete full-dataset validations for NYBG & SALVIAS
- Created mechanism to write verification queries and compare to desired output
- 04:57 PM Task #304: Complete full dataset imports to VegBIEN via VegX of NYBG and SALVIAS
- Core SALVIAS data (plots and organisms) are on vegbiendev in VegBIEN.
Takes overnight to reinsert organisms data whe... - 04:54 PM Revision 372: input.Makefile: Don't remove logs in clean
- 04:49 PM Task #303: Mapping from VegBIEN to original VegBank
- Search-and-replace operations that transform VegBank to VegBIEN can in many cases be reversed to transform VegBIEN to...
- 04:48 PM Task #302: Make changes to VegBIEN schema
- All functions that used to output to VegBank now output to VegBIEN db.
The VegBIEN schema is automatically generated... - 04:45 PM Task #296: Direct mapping from native salvias_plots MySQL database to VegBIEN
- We will also need to import the projects, stems, and lookup_MethodCode tables in order for all the verification queri...
- 04:43 PM Task #290: benchmark tests for database loading
- Created mechanism to write verification queries and compare to desired output
- 04:32 PM Revision 371: input.Makefile: Added support for multi-column verification queries. inputs/SALVIAS: Added a multi-column verification query.
- 03:47 PM Revision 370: input.Makefile: Remove all *.out in clean
- 02:59 PM Revision 369: inputs: Don't keep *.out under version control
- 02:35 PM Revision 368: input.Makefile: Added rules to verify inserted data and run tests
- 11:38 AM Revision 367: Regenerated inputs/SALVIAS/map.VegBIEN.1.organisms.csv
- 11:32 AM Revision 366: exc.py: Print ! before error messages so they can be found and counted more easily in log files
- 11:13 AM Revision 365: input.Makefile: Only output to log if importing all rows
- 10:46 AM Revision 364: Moved vegbien CREATE DATABASE command to Makefile and run vegbien.sql as bien user
- 10:27 AM Revision 363: Renamed vegbien tables according to Bob Peet's new VegBank-VegBIEN mapping
- 10:16 AM Revision 362: input.Makefile: Don't print commands that use set -x to be echoed
- 10:14 AM Revision 361: input.Makefile: Use system's time utility instead of shell's time command
- 10:09 AM Revision 360: sql.py: Include sql.DbException in DatabaseErrors list
01/03/2012
- 04:28 PM Revision 359: Generate VegX-VegBIEN.organisms.csv from VegBank-VegBIEN.map.csv, which is generated from VegBank-VegBIEN.map_only.csv
- 04:21 PM Revision 358: repl: Process whole file as one string so that regexps can match EOF for appending
- 04:21 PM Revision 357: mappings/Makefile: Clean *-VegBIEN.*.csv maps
- 04:08 PM Revision 356: Generate VegBank-VegBIEN.schema.csv from VegBank-VegBIEN.schema_only.csv and VegBank-VegBIEN.csv
- 04:06 PM Revision 355: Added cat_csv to concatenate two spreadsheets
- 03:34 PM Revision 354: Removed no longer needed inputs/SALVIAS/db.sh
- 03:24 PM Revision 353: Add sort order prefixes to inputs/SALVIAS maps
- 02:49 PM Revision 352: VegX-VegBank organisms mapping: Added collectionDate mapping
- 02:36 PM Revision 351: VegX-VegBank organisms mapping: Added collectionDate mapping
- 02:36 PM Revision 350: mappings/Makefile: Sort VegX-VegBank organisms mapping automatically when anything that depends on it is remade
- 02:34 PM Revision 349: Added in_place to run a stream command on a file
- 02:14 PM Revision 348: join_sort: Use /bin/sh instead of /bin/bash
- 01:44 PM Revision 347: NYBG-VegX mapping: Removed no longer needed global stem count value
- 01:42 PM Revision 346: Renamed test/accept_output to accept
- 01:40 PM Revision 345: VegX-VegBank.organisms.csv: Made stemCount default to 1
- 01:17 PM Revision 344: bin/map: Print DatabaseErrors without stack traces
- 01:13 PM Revision 343: exc.py: Added print_ex()
- 01:13 PM Revision 342: exc.py: Added print_ex()
- 01:13 PM Revision 341: util.py: Added type_name()
- 01:12 PM Revision 340: strings.py: Added ensure_newl()
- 11:42 AM Revision 339: input.Makefile: Save map output in log file
- 11:27 AM Revision 338: bin/map: Print usage message when run with no arguments
- 10:10 AM Revision 337: Makefile: Removed unnecessary commenting out of "allow from 127.0.0.0..." lines in /etc/phppgadmin/apache.conf
12/29/2011
- 05:35 PM Revision 336: Makefile: Use --loose-exit-status for ed to avoid failing on no match. In editPhppgadminApacheConf, use ,s/// instead of /.../\ns/// to match lines regardless of position.
- 05:17 PM Revision 335: Makefile: Use tee instead of ed in sudoAppend. Don't change apache port to 8080 if apache already running on 80.
- 04:54 PM Revision 334: Makefile: Added phppgadmin apache config
- 01:16 PM Revision 333: bin/map: Only treat DatabaseErrors as warnings, not all Exceptions
- 01:04 PM Revision 332: xml_func.py: Print newline before func XML because XML is now pretty-printed with indents
- 12:59 PM Revision 331: xml_dom.py: Node.__str__() uses toprettyxml() instead of toxml()
- 12:58 PM Revision 330: util.py: Changed rename_key() not to suppress KeyError, since some callers might want to assert that a key exists
- 12:44 PM Revision 329: Makefile: Added phpPgAdmin dependency
- 12:42 PM Revision 328: xml_func.py: _name func doesn't require all name parts to be set. _namePart func throws SyntaxException for invalid name parts.
12/28/2011
- 02:03 PM Revision 327: bin/map: Fixed syntax error where db variable had not been renamed to out_db
- 01:58 PM Revision 326: xml_func.py: Generate SyntaxExceptions on missing XML func args
- 01:46 PM Revision 325: VegX-VegBank mapping: Added default soilObs.soilHorizon value
- 01:33 PM Revision 324: xml_func.py: Changed _date func to try parsing date field, rather than year field, as a decimal. Changed SALVIAS mappings to use new _date func syntax. Fixes SALVIAS plots data syntax error for decimal years.
- 01:10 PM Revision 323: Removed no longer needed VegBank accepted test outputs
- 01:09 PM Revision 322: test/map: Abort tester on failed diff
- 01:09 PM Revision 321: Added accepted test output SALVIAS_db.organisms.VegBIEN.xml
- 01:04 PM Revision 320: SALVIAS organisms mapping: Use _alt to merge tag1 and tag2. Fixes failing diff due to duplicate stemCodes.
- 12:49 PM Revision 319: test/map: Added tests to insert VegBIEN XML into VegBIEN db
- 12:47 PM Revision 318: bin/map: Fixed non-map input to set out_label
- 12:12 PM Revision 317: bin/map: Fixed row processing when no map is used
- 12:06 PM Revision 316: bin/map: Refactored to process each row separately when out_is_db
- 12:04 PM Revision 315: xml_dom.py: Added Node pop() and clear() methods
- 11:15 AM Revision 314: bin/map: Moved process_rows() exception handling around entire row, rather than each value, so that a broken value will not be ignored only to cause errors when outputing the row
- 11:12 AM Revision 313: bin/map: Moved process_rows() exception handling around entire row, rather than each value, so that a broken value will not be ignored only to cause errors when outputing the row
- 11:05 AM Revision 312: db_xml.py: Removed no longer needed xml2db()
- 11:04 AM Revision 311: bin/map: out_is_db uses use_row instead of xml2db()
- 10:53 AM Revision 310: bin/map: Added commit and exception-handling code to out_is_db use_row()
- 10:42 AM Revision 309: bin/map: Merged get_xml() into process_xml(). Renamed process*() functions to distinguish them better.
- 10:33 AM Revision 308: bin/map: Create separate document for printing XML opening and closing
- 10:28 AM Revision 307: bin/map: Refactored to print document opening, contents, and closing in separate steps so they can be separated later
- 10:27 AM Revision 306: xml_dom.py: Added support for integer indents
- 10:19 AM Revision 305: xml_dom.py: Added NodeIter() and Node.__iter__()
- 10:05 AM Revision 304: Use new xml_dom.prettyxml_config instead of Node.writeprettyxml because it can be passed as keyword args to any writexml function
- 10:00 AM Revision 303: xml_dom.py: Removed no longer needed xml_dom.writexml()
- 10:00 AM Revision 302: bin/map: Use Node.writeprettyxml() instead of xml_dom.writexml()
- 09:57 AM Revision 301: xml_dom.py: Added minidom.Document write_opening() and write_closing() methods. Changed writexml(writer, node) to use new Node.writeprettyxml() method.
- 09:28 AM Revision 300: Renamed ex.py to exc.py
- 09:24 AM Revision 299: Import xml.dom.minidom as minidom
- 09:10 AM Revision 298: xml_dom.py: Refactored xml.dom.minidom.Element.writexml() to use new write_opening() and write_closing() methods. Use new functions is_elem() and is_text_node() to determine node type. Corrected definition of a text element to be an element that contains a single text node.
- 08:37 AM Revision 297: bin/map: Factored out input processing code
12/23/2011
- 04:03 PM Revision 296: Commit after inserting each row
- 03:59 PM Revision 295: Commit after inserting each row
- 03:54 PM Revision 294: Use traceback.print_exc() to print Exceptions for each row instead of exiting
- 03:26 PM Revision 293: env_password: Allow user to change saved password
- 03:14 PM Revision 292: env_password: Don't print instructions for storing password for the session if called *to* store password for the session
- 03:00 PM Revision 291: env_password: Print absolute path to env_password in case invoking script changed the working directory
- 02:57 PM Revision 290: env_password: Print instructions for storing password for the session
- 02:32 PM Revision 289: Updated SALVIAS plots mappings and corresponding VegX-VegBank mappings to parse date_start and date_finish
- 02:11 PM Revision 288: xml_func.py: Print XML of XML functions with SyntaxExceptions
- 02:08 PM Revision 287: xml_func.py: Added support for parsing dates
- 01:56 PM Revision 286: ex.py: repl_msg(): take replacements as keyword args
- 01:44 PM Revision 285: fix_permissions: Also change user to bien
- 01:31 PM Revision 284: Makefile: Added python-dateutil dependency
- 12:57 PM Revision 283: input.Makefile: Renamed from_db-<map_file> target to import-<type> so it can be more easily run as a command-line target rather than just a prerequisite
- 12:40 PM Revision 282: Added inputs/SALVIAS organisms map
12/22/2011
- 08:30 PM Revision 281: map: Print xml_func.SyntaxExceptions without stack traces by using SystemExit
- 08:22 PM Revision 280: xml_func.py: Add function name to SyntaxException message
- 08:22 PM Revision 279: ex.py: Added repl_msg() to format a message with the % operator
- 07:48 PM Revision 278: xml_func.py: Return string->number conversion errors as xml_func.SyntaxExceptions
- 07:29 PM Revision 277: psql_vegbien: Fixed comment to vegbien instead of vegbank
- 07:11 PM Revision 276: psql_vegbien: Use new location of bien_password
- 07:05 PM Revision 275: Makefile: Fixed paths to mappings dir for new scripts dir location
- 07:02 PM Revision 274: Renamed util to bin
- 06:59 PM Revision 273: Moved inputs_Makefile to inputs/input.Makefile
- 06:55 PM Revision 272: Moved bien_password to new config dir
- 06:52 PM Revision 271: Moved sample inputs to test dir
- 06:42 PM Revision 270: Added symlink from scripts to new scripts destination
- 06:40 PM Revision 269: Removed now-empty scripts dir
- 06:39 PM Revision 268: Moved everything in scripts to root. inputs_Makefile: Don't run "all" when installing.
- 06:24 PM Revision 267: Renamed bien_map to map
- 06:22 PM Revision 266: Moved map to util
- 06:14 PM Revision 265: fix_permissions: Don't chmod symlinks
- 06:00 PM Revision 264: inputs_Makefile: Auto-generate map to VegBIEN and import data into vegbien from input DB
- 05:59 PM Revision 263: inputs/SALVIAS: maps to VegX and VegBIEN
- 05:58 PM Revision 262: bien_map: Runs map with BIEN defaults
- 05:56 PM Revision 261: join_sort: Sorts a join on the output col
- 03:20 PM Revision 260: inputs_Makefile: Require dbEngine var instead of defaulting to MySQL
- 02:22 PM Revision 259: Moved inputs into svn
- 01:46 PM Revision 258: Moved pre-BIEN 3 files into _archive folder
- 01:46 PM Revision 257: test/map: Use db.sh syntax in *.sh tests
- 01:45 PM Revision 256: inputs_Makefile: Generate db.sh with DB access info
- 12:59 PM Revision 255: inputs_Makefile: Changed GRANT ALL to GRANT SELECT. Added REVOKE ALL. Added $(db).sql as prerequisite of install in case it needs to be auto-generated.
- 12:41 PM Task #302: Make changes to VegBIEN schema
- Bob Peet's changes are at [[VegBIEN from VegBank]]
E-mail from Bob Peet on 2011-12-22:
I promised to summarize ... - 12:31 PM Task #310 (Resolved): automated build process
- I set up an automated build process for all the BIEN scripts and dependencies. It runs on both vegbiendev and nimoy. ...
- 12:31 PM Task #310 (Resolved): automated build process
- 12:30 PM Revision 254: scripts/Makefile: Added action for postgres-Darwin target
- 12:11 PM Revision 253: scripts/Makefile: Added postgresql to postgres-Linux apt-get packages
12/21/2011
- 08:23 PM Revision 252: test/map: Changed to work on both nimoy and vegbiendev by selecting the appropriate MySQL user and password
- 08:22 PM Revision 251: env_password: Added optional message arg
- 07:47 PM Revision 250: Added uninstallation of inputs to Makefiles
- 07:15 PM Revision 249: scripts/Makefile: Create bien user w/o prompting for password. Fixed syntax error.
- 07:06 PM Revision 248: scripts/Makefile: Fixed syntax error
- 07:03 PM Revision 247: Inputs now include inputs_Makefile to get mysql command, etc.
- 06:30 PM Revision 246: scripts/Makefile: Use bien MySQL user for installing inputs
- 06:26 PM Revision 245: scripts/Makefile: Use root MySQL user for creating bien user
- 05:53 PM Revision 244: scripts/Makefile: Don't use root as MySQL admin user. Removed no-longer-needed sub-makes for setting DB login vars.
- 05:10 PM Revision 243: Changed vegbien_dest and users of it to use separate bien_password file
- 04:55 PM Revision 242: fix_permissions: Extend all user permissions to group
- 04:26 PM Revision 241: scripts/Makefile: Added inputs
- 03:24 PM Revision 240: mappings/Makefile: Cleaned up
- 03:02 PM Revision 239: fix_permissions: Don't add group write perms to read-only files
- 01:54 PM Revision 238: scripts/Makefile: Made read command syntax compatible with /bin/sh
- 01:37 PM Revision 237: scripts/Makefile: Ignore errors about missing packages
- 01:30 PM Revision 236: scripts/Makefile: Fixed escape character for /bin/sh
- 01:25 PM Revision 235: scripts/Makefile: Added dependency installation. Makefiles: Use _not_file instead of FORCE for clarity. Use $(os) var
12/20/2011
- 07:52 PM Revision 234: fix_permissions: Configured output verbosity
- 07:44 PM Revision 233: Added fix_permissions to set correct permissions on shared bien files
- 07:31 PM Revision 232: Removed VegBank scripts which are no longer needed
- 07:30 PM Revision 231: Removed VegBank scripts which are no longer needed
- 06:55 PM Revision 230: Renamed vegbien_dest.sh to vegbien_dest to reflect that it is also includable by Makefiles
- 06:52 PM Revision 229: DB user creation: Clarified instructions
- 06:51 PM Revision 228: scripts/Makefile includes vegbien_dest.sh directly
- 06:28 PM Revision 227: Moved db user creation to scripts/Makefile. Removed now-unneeded admin scripts.
- 06:12 PM Revision 226: Removed VegBank scripts which are no longer needed
- 06:11 PM Revision 225: scripts/Makefile: Added empty_db target which uses vegbien_empty.sql
- 05:55 PM Revision 224: mappings/Makefile: Fixed bug where var containing prerequisistes needed to be defined before used. Added support for different sed flags to use extended regular expressions.
- 05:42 PM Revision 223: Added auto-generated vegbien_empty.sql to empty the vegbien db
- 04:55 PM Revision 222: Test output to VegBIEN instead of VegBank
- 04:54 PM Revision 221: VegX-VegBIEN.organisms.csv: sort output of repl
- 04:46 PM Revision 220: review: Added nullglob
- 04:44 PM Revision 219: review: Don't process replacements spreadsheets
- 04:44 PM Revision 218: Moved schema replacements from VegBank-VegBIEN.csv to VegBank-VegBIEN.schema.csv
- 04:32 PM Revision 217: VegBank-VegBIEN.csv: Support PostgreSQL before 8.4
- 04:29 PM Revision 216: repl: Added support for blank lines. Only add whole word regexp code to inputs w/o *any* regexp metachars.
- 04:13 PM Revision 215: Create vegbien db from mappings/schemas/vegbien.sql
- 04:12 PM Revision 214: VegBank-VegBIEN.csv: Added replacements for SQL create script conversion
- 03:56 PM Revision 213: Generate vegbien db create SQL from vegbank.sql using repl
- 03:55 PM Revision 212: Generate vegbien db create SQL from vegbank.sql using repl
- 03:54 PM Revision 211: repl: Fixed bug in reading arguments
- 03:38 PM Revision 210: repl: Don't add whole-word regexp for inputs that already have regexp metachars
- 03:26 PM Revision 209: Removed mappings/VegBank-VegBIEN.organisms.csv because now using replacements spreadsheet
- 03:24 PM Revision 208: mappings: Generate mappings to VegBIEN using VegBank-VegBIEN.cs replacements spreadsheet
- 03:23 PM Revision 207: mappings: Generate mappings to VegBIEN using VegBank-VegBIEN.cs replacements spreadsheet
- 03:23 PM Revision 206: mappings: Generate mappings to VegBIEN using VegBank-VegBIEN.cs replacements spreadsheet
- 02:57 PM Revision 205: Added repl to perform replacements on a spreadsheet or file
- 01:38 PM Revision 204: scripts/Makefile: removed extra test-% target
- 01:38 PM Revision 203: README.TXT: Updated
- 01:36 PM Revision 202: scripts/Makefile: install/uninstall targets
- 01:36 PM Revision 201: bien_user_create: Print instructions in sequence with password prompts
12/19/2011
- 06:16 PM Revision 200: mappings to VegBIEN: Accounted for remaining ALTER TABLE statements
- 05:48 PM Revision 199: Renamed format*_for_review to review and added for_review to make clean
- 05:41 PM Revision 198: mappings: Added mappings to VegBIEN
- 05:25 PM Revision 197: mappings/Makefile: Simplified
- 05:22 PM Revision 196: mappings/Makefile: Simplified
- 04:59 PM Revision 195: README.TXT: Updated
- 04:47 PM Revision 194: Added vegbien DB admin scripts
12/16/2011
- 03:54 PM Revision 193: join_all_vegbank: Sort output by both columns
- 03:48 PM Revision 192: mappings/VegX-VegBank.organisms.csv: Sorted by both columns
- 03:48 PM Revision 191: mappings/Makefile: Sort VegBank-VegBIEN.organisms.csv by both columns
- 03:47 PM Revision 190: extract_plot_map: Removed because functionality now in Makefile
- 03:46 PM Revision 189: sort: Can sort on multiple columns
- 03:17 PM Revision 188: Added basic VegX-VegBIEN mapping
- 02:02 PM Revision 187: Added union and join_passthru
- 02:02 PM Revision 186: env_password: Print Usage message to stderr
- 01:29 PM Revision 185: test/map: Create output dir if it doesn't exist
- 01:24 PM Revision 184: Converted scripts back to bash that required bash-specific features
- 01:23 PM Revision 183: Converted scripts back to bash that required bash-specific features
- 01:13 PM Revision 182: Fixed test/map to work with sh
- 01:05 PM Revision 181: Replaced /bin/bash with /bin/sh
- 12:31 PM Task #300: TurboVeg data
- Mike Lee's mapping is on nimoy under @/home/bien_shared/raw_data/turboveg/DBASEDIC_rkp2011_mtl2011.xlsx@
- 12:21 PM Task #309 (Rejected): mapping and export utility from VegBank to VegX
- Ideally, what I have in mind is mapping and export utility from VegBank to VegX. Of course this means more work up fr...
- 12:20 PM Task #308 (Resolved): do a direct transfer of some public data from VegBank
- Even higher priority, do you think you could set us up to do a direct transfer of some public data from VegBank? At t...
12/15/2011
- 04:16 PM Revision 180: join: Added usage item for repeated joins
- 04:13 PM Revision 179: join: Changed order of args and redirects to be more intuitive
- 04:09 PM Revision 178: Renamed ch_map_root to ch_root
- 04:08 PM Revision 177: Renamed join_maps to join
- 03:42 PM Task #307 (Resolved): Acquire additional specimen data sets in both DwC and DwCA format, esp. GBIF
- 03:42 PM Task #306 (Resolved): Acquire additional plot data sets from providers willing to work with Aaron on mappings and validations
- Bob: TurboVeg; Brad: RAINFOR, CTFS
- 03:42 PM Task #305 (Resolved): Complete full-dataset validations for NYBG & SALVIAS
- 03:41 PM Task #304 (Resolved): Complete full dataset imports to VegBIEN via VegX of NYBG and SALVIAS
- * Identify and make changes to VegX needed to enable full-dataset imports
* Or report changes needed to Nick, Miquel... - 03:41 PM Task #303 (New): Mapping from VegBIEN to original VegBank
- the latter to be used as web schema for BIEN web interface
- 03:40 PM Task #302 (Resolved): Make changes to VegBIEN schema
- 02:49 PM Task #294: find plot data source provider to work with Aaron
- Brad has contacted two data source provider (RAINFOR, CTFS) regarding working with Aaron to develop mappings
- 01:55 PM Task #285 (Resolved): CSV to XML mappings for NYBG, SALVIAS
- initial data sources NYBG and SALVIAS have been fully mapped
- 01:53 PM Task #291 (Resolved): list of milestones
- got commented milestones from Martha
- 01:03 PM Task #286: CSV-XML-database mapping script
- I added support for mapping XML to XML, which will enable us to process NVS's VegX data, and eventually also their in...
- 01:03 PM Task #296: Direct mapping from native salvias_plots MySQL database to VegBIEN
- I added a new mapping to get SALVIAS data directly from the salvias_plots database on nimoy. You can see the results ...
- 12:20 PM Revision 176: Merged test Makefile into main scripts Makefile
- 12:05 PM Revision 175: test/map: Turn off test mode (don't run diff) when env var n (for # rows) is set
- 11:39 AM Revision 174: Added SALVIAS DB mapping for plots data
- 11:37 AM Revision 173: VegX-VegBank mapping: Fixed VegBank XPath for commName.commName field
- 11:35 AM Revision 172: db_xml.py: Use pointer target's name as pointer type where possible. Emphasize that pointer type determined from the pointer name itself is a guess based on common database conventions.
- 11:31 AM Revision 171: xpath.py: Changed backward (child-to-parent) pointer ID abbr expansion to happen in get() when source node's tag name is known. This deals with XPath elements that are '.' being used as a pointer source.
12/14/2011
- 05:31 PM Revision 170: xpath.py: Moved abbr expansion code to separate function
- 04:41 PM Revision 169: test/map: Process all tables for a given DB (.sh) input
- 03:26 PM Revision 168: Removed /'s from DB input mappings
- 02:29 PM Task #299: Mapping from NVS to VegX and VegBIEN
- NVS data from Nick Spencer is on nimoy in @/home/bien_shared/raw_data/nvs/@
- 02:06 PM Task #299 (Resolved): Mapping from NVS to VegX and VegBIEN
- Will require finding someone at NVS willing to work with Aaron and mappings and validations
- 02:25 PM Task #300: TurboVeg data
- TurboVeg info from Bob Peet is on nimoy in @/home/bien_shared/raw_data/turboveg/@
- 02:07 PM Task #300 (New): TurboVeg data
- with commitment by someone familiar with DB to work with Aaron on (a) evaluating mappings, and (b) developing validat...
- 02:07 PM Task #301 (Resolved): RAINFOR data
- with commitment by someone familiar with DB to work with Aaron on (a) evaluating mappings, and (b) developing validat...
- 02:06 PM Task #298 (New): Try to find source of DwCA (DwC Archives) data
- hoping GBIF will be willing to work with us on this. Possibly approach Remsen directly
- 02:05 PM Task #297 (Resolved): Request new data dump of specimen data from GBIF, this time in DwC format
- 02:03 PM Task #296 (Resolved): Direct mapping from native salvias_plots MySQL database to VegBIEN
- 01:59 PM Task #289: look for formal mapping mechanism
- Got NVS mapping tool from Nick Spencer, which is on nimoy in @/home/bien_shared/raw_data/nvs/VegX/@
- 01:54 PM Task #286: CSV-XML-database mapping script
- Added support for database and XML inputs
- 01:46 PM Revision 167: map: Use row's index instead of pkey as ID in XML output
- 01:45 PM Revision 166: test/map: Compare via-VegX output to direct output
- 01:13 PM Revision 165: xpath.py: Changed order that main and other branches are processed in so it is consistent with the order the branches are specified in the XPath
- 12:03 PM Revision 164: map: Handle metadata in order with regular mappings
- 11:33 AM Revision 163: Accepted VegBank test output for new CSV mapping order
- 11:26 AM Revision 162: map: Changed CSV input to process mappings in the order they are in the spreadsheet, rather than the order of the CSV columns
12/13/2011
- 05:18 PM Revision 161: map: Added support for XML input
- 05:17 PM Revision 160: Accepted new test output for sorted SALVIAS_db-VegBank mapping
- 05:07 PM Revision 159: mappings to VegBank: Sorted by output column to help VegX-VegBank conversion put elements in the same order as source-VegBank
- 04:57 PM Revision 158: join_all_vegbank: Updated to sort output maps
- 04:57 PM Revision 157: Added script to sort a spreadsheet
- 03:51 PM Revision 156: xpath.py: Allowed empty names in XPaths
- 03:48 PM Revision 155: xpath.py: Added automatic conversion of strings to paths where needed.
- 03:00 PM Revision 154: xpath.py: Added caching of parsed XPaths. Added automatic conversion of strings to paths where needed.
- 02:59 PM Revision 153: Added __str__() method to XML nodes
- 02:58 PM Revision 152: Fixed VegX-VegBank mapping syntax error
- 02:17 PM Revision 151: Added faded beginning of string in Parser syntax errors
- 02:01 PM Revision 150: Updated mappings Makefile
- 02:00 PM Revision 149: Added Makefiles for scripts and test
- 01:54 PM Revision 148: Added mappings Makefile
- 12:28 PM Revision 147: Added human-readable SALVIAS_db mappings
- 12:28 PM Revision 146: db_xml.get(): Pass limit through to SQL query
- 12:11 PM Task #295 (Resolved): provide benchmark queries for NYBG data
- Brad Boyle provided NYBG queries, which are on the wiki under [[NYBG tests]]
- 12:10 PM Task #290: benchmark tests for database loading
- Brad Boyle provided NYBG queries, which are on the wiki under [[NYBG tests]]
- 11:57 AM Task #291: list of milestones
- updated to do list
- 11:56 AM Task #291: list of milestones
- Brad created a timeline, which is on the wiki under [[December 8 2011 WebEx meeting]].
- 11:48 AM Revision 145: Regenerated human-readable mappings
12/12/2011
- 05:39 PM Revision 144: Fixed documentation for xml_funcs
- 05:38 PM Revision 143: Refactored xml_dom.set_value() to avoid needing a doc parameter for the XML document
- 05:35 PM Revision 142: xpath.py: Refactored xml_func.py to avoid needing a doc parameter for the XML document
- 05:30 PM Revision 141: xpath.py: Refactored to avoid needing a doc parameter for the XML document
- 04:41 PM Revision 140: Fixed DB input to ignore NULL values
- 04:27 PM Revision 139: xml_dom.py: Changed all uses of name_of(node) to node.tagName
- 04:23 PM Revision 138: Made XML node names case-sensitive
- 04:20 PM Revision 137: mappings to VegBank: Fixed incorrect mappings found after disabling heuristic search for missing fields
- 03:49 PM Revision 136: test/map: Ignore diff exit status
- 03:34 PM Revision 135: map: Implemented DB input support for querying a single table
Also available in: Atom