Project

General

Profile

Activity

From 11/06/2011 to 12/05/2011

12/05/2011

05:25 PM Revision 93: Started filling in required values for VegBank fields in mappings. Will need to refactor to move these to metadata for the datasources.
Aaron Marcuse-Kubitza
05:24 PM Revision 92: Now allow empty rows. Added support for select statement limit.
Aaron Marcuse-Kubitza
04:17 PM Revision 91: Added support for quoted values in XPaths
Aaron Marcuse-Kubitza
04:02 PM Revision 90: Fixed name XML function. Fixed accept_test_output.
Aaron Marcuse-Kubitza
03:59 PM Revision 89: Added support for name XML function. Added error handling for empty rows.
Aaron Marcuse-Kubitza
03:28 PM Revision 88: Made it easier to accept test output
Aaron Marcuse-Kubitza
03:18 PM Revision 87: Added NYBG stemCount metadata
Aaron Marcuse-Kubitza
03:11 PM Revision 86: Added xml_func.py to process mappings whose output needs postprocessing
Aaron Marcuse-Kubitza
01:53 PM Revision 85: Changed VegBank mappings to use XML functions (not implemented yet) to calculate averages and ranges
Aaron Marcuse-Kubitza
01:25 PM Revision 84: Added support for mapping datasource metadata
Aaron Marcuse-Kubitza
12:53 PM Revision 83: Changed for loops to use enumerate() where the index is also needed
Aaron Marcuse-Kubitza
12:50 PM Revision 82: Moved XPath prep code (setting ID, value) to xpath.py
Aaron Marcuse-Kubitza
12:14 PM Task #292: VegBank metadata query mechanism
(Moved to issue description) Aaron Marcuse-Kubitza
12:13 PM Task #292 (New): VegBank metadata query mechanism
For data discovery of VegBank schema.
Mike Lee's suggestion: (e-mail on 2011-11-9)
I'm wondering if you all tal...
Aaron Marcuse-Kubitza
12:09 PM Task #289: look for formal mapping mechanism
Mike Lee's explanation of the VegBank XML serialization format: (e-mail on 2011-12-2)
My recollection is that our in...
Aaron Marcuse-Kubitza

12/02/2011

05:27 PM Revision 81: xpath.py: Added deepcopy() before setting value of other branches to traverse
Aaron Marcuse-Kubitza
05:12 PM Revision 80: NYSpecimenDataAmericas.test.xml: Updated for new NYBG-VegX.organisms.csv
Aaron Marcuse-Kubitza
05:11 PM Revision 79: NYBG-VegX.organisms.csv: Changed voucher (primary key) column to be UniqueNYInternalRecordNumber because CatalogNumber contained an empty value
Aaron Marcuse-Kubitza
05:10 PM Revision 78: xpath.py: Added basic support for split paths
Aaron Marcuse-Kubitza
04:30 PM Revision 77: Merged xml_xpath.py into xpath.py in preparation for changing the XPath parse tree to be the XML DOM tree itself
Aaron Marcuse-Kubitza
03:58 PM Revision 76: Refactored xpath.parse() to use a nested function instead of a class extending Parser
Aaron Marcuse-Kubitza
03:04 PM Revision 75: map: Fixed mislocated import for Parser.SyntaxException
Aaron Marcuse-Kubitza
02:21 PM Revision 74: Removed SALVIAS voucher_string mapping per conference call discussion
Aaron Marcuse-Kubitza
02:16 PM Revision 73: map: Fixed bugs to enable mapping straight from CSV to a database. Still need a way to set plot.authorPlotCode for specimens data.
Aaron Marcuse-Kubitza
12:05 PM Revision 72: Fixed ch_map_root to support subpaths which follow the root by -> rather than /. Changed spreadsheet syntax to have : between label and root.
Aaron Marcuse-Kubitza

12/01/2011

04:44 PM Task #291: list of milestones
I put the tasks from today's conference call into the "Redmine issue tracker":https://projects.nceas.ucsb.edu/nceas/p... Aaron Marcuse-Kubitza
04:35 PM Task #291 (Resolved): list of milestones
*Conference call:*
* -need list of milestones for the next 6-12 months-
* -*add conference call tasks to Redmine ...
Aaron Marcuse-Kubitza
04:34 PM Task #290 (Resolved): benchmark tests for database loading
*Conference call:*
* *develop benchmark tests to check that datasource data was inserted correctly into VegBank*
...
Aaron Marcuse-Kubitza
04:32 PM Task #289 (Resolved): look for formal mapping mechanism
*Conference call:*
* look into VegBranch's way of capturing mappings and metadata
* -look into Altova XMLSpy's gr...
Aaron Marcuse-Kubitza
04:29 PM Task #285: CSV to XML mappings for NYBG, SALVIAS
*Conference call:*
* Ignore SALVIAS @voucher_string@ because it is sometimes missing collector's name
* Ignore SALVI...
Aaron Marcuse-Kubitza
10:50 AM Task #285: CSV to XML mappings for NYBG, SALVIAS
The latest mapping spreadsheets for datasources->VegX and VegX->VegBank are now available in svn at https://projects.... Aaron Marcuse-Kubitza
01:55 PM Revision 71: Updated extract_plot_map to use new name for VegX-VegBank mapping and re-ran it and join_all_vegbank
Aaron Marcuse-Kubitza
01:51 PM Revision 70: Finished VegX-VegBank mapping and created VegBank joins of mappings to VegX
Aaron Marcuse-Kubitza
11:53 AM Revision 69: Finished ch_map_root (renamed from submap)
Aaron Marcuse-Kubitza
10:53 AM Task #286: CSV-XML-database mapping script
I've merged data2xml and xml2db into one script called map, which can be run on nimoy at /home/bien_shared/svn/script... Aaron Marcuse-Kubitza
10:52 AM Task #288: VegX-VegBank mapping
The mapping spreadsheet for VegX->VegBank is now available in svn at https://projects.nceas.ucsb.edu/nceas/projects/b... Aaron Marcuse-Kubitza

11/30/2011

05:36 PM Revision 68: Added submap and extract_plot_map to extract plot subpaths from VegX-VegBank.csv
Aaron Marcuse-Kubitza
04:56 PM Revision 67: Moved env usage string creation to opts.py. Changed db config var names to use in/out instead of from/to.
Aaron Marcuse-Kubitza
04:24 PM Revision 66: Keep *.test.xml out of version control
Aaron Marcuse-Kubitza
04:22 PM Revision 65: Moved options-processing code to opts.py: Added opts.py
Aaron Marcuse-Kubitza
04:21 PM Revision 64: Moved options-processing code to opts.py
Aaron Marcuse-Kubitza
04:04 PM Revision 63: test_map: Compares generated XML to correct version
Aaron Marcuse-Kubitza
03:55 PM Revision 62: Fixed xml_xpath.get() last_only optimization to handle attrs correctly. Turned off stack traces for errors intended for the user to see.
Aaron Marcuse-Kubitza
02:32 PM Revision 61: Changed mappings to place prefix common to all XPaths in the column header
Aaron Marcuse-Kubitza
01:40 PM Task #288 (Resolved): VegX-VegBank mapping
CSV spreadsheet mapping VegX to VegBank Aaron Marcuse-Kubitza
01:37 PM Task #287: XML to database conversion script (merged into CSV-XML-database mapping script)
Merged into CSV-XML-database mapping script Aaron Marcuse-Kubitza
01:35 PM Task #286: CSV-XML-database mapping script
Merged in XML to database conversion script Aaron Marcuse-Kubitza
01:31 PM Revision 60: simplify_xpath: Made it case-insensitive
Aaron Marcuse-Kubitza
01:25 PM Revision 59: map: Added support for custom fkeys to parent in db XML trees. Removed extraneous csv reader/writer config because Excel format is default. Improved documentation.
Aaron Marcuse-Kubitza

11/29/2011

05:36 PM Revision 58: map: Added stub for database input
Aaron Marcuse-Kubitza
05:33 PM Revision 57: map: Added more stubs for XML-XML mapping
Aaron Marcuse-Kubitza
05:15 PM Revision 56: Started adding XML-XML mapping support to map
Aaron Marcuse-Kubitza
04:43 PM Revision 55: Split off xpath.py XML functionality into xml_xpath.py
Aaron Marcuse-Kubitza
04:28 PM Revision 54: map: Using SystemExit for usage errors to avoid stack trace
Aaron Marcuse-Kubitza
04:22 PM Revision 53: Merged data2xml and xml2db into map
Aaron Marcuse-Kubitza
03:03 PM Revision 52: Removed trailing whitespace from VegX-VegBank.csv map
Aaron Marcuse-Kubitza
02:59 PM Revision 51: Created join_maps to join two 2-column map spreadsheets
Aaron Marcuse-Kubitza
02:11 PM Revision 50: Renamed mappings to be compatible with Redmine allowed characters in attachment filenames
Aaron Marcuse-Kubitza
01:59 PM Revision 49: Added refactored mappings and changed data2xml to use the new 2-column format
Aaron Marcuse-Kubitza
01:25 PM Revision 48: Refactored db_xml.py's db insertion function to avoid extra nested functions
Aaron Marcuse-Kubitza
01:06 PM Revision 47: Added README.TXT
Aaron Marcuse-Kubitza
01:02 PM Revision 46: Renamed modules to remove _util
Aaron Marcuse-Kubitza
12:47 PM Revision 45: Added svn:ignore for *.pyc
Aaron Marcuse-Kubitza
12:42 PM Revision 44: Renamed xml2db_ and data2xml_ to remove _
Aaron Marcuse-Kubitza
12:42 PM Revision 43: Moved scripts to main directory and associated files to util
Aaron Marcuse-Kubitza
12:31 PM Revision 42: Moved Python modules to shared lib folder
Aaron Marcuse-Kubitza

11/28/2011

05:32 PM Revision 41: xml2db: Started refactoring xml2db() to support getting as well as inserting data
Aaron Marcuse-Kubitza
05:29 PM Revision 40: xml2db: Started refactoring xml2db() to support getting as well as inserting data
Aaron Marcuse-Kubitza
05:05 PM Revision 39: xml2db: Changed to return ID (pkey) of inserted record and use this returned value as parent_id instead of getting the parent_id from the parent XML node
Aaron Marcuse-Kubitza
03:16 PM Revision 38: data2xml: Added syntax for split paths, which map to multiple leaves
Aaron Marcuse-Kubitza
01:52 PM Revision 37: xml2db: Improved empty_db to use TRUNCATE instead of DROP DATABASE. Added xml2vegbank to automatically set db env vars.
Aaron Marcuse-Kubitza
01:51 PM Revision 36: data2xml: Improved syntax for XPath lookahead assertions. Changed XML printing to print multiple text nodes on separate lines.
Aaron Marcuse-Kubitza
12:15 PM Revision 35: Moved vegbank_example_ver1.0.2.xml to xml2db, where it should have been
Aaron Marcuse-Kubitza

11/23/2011

05:39 PM Task #286: CSV-XML-database mapping script
I updated the data2xml script (demo at nimoy:/home/bien_shared/svn/scripts/data2xml/test) to support the pointer form... Aaron Marcuse-Kubitza
05:37 PM Task #285: CSV to XML mappings for NYBG, SALVIAS
I made a number of changes to the NYBG and SALVIAS mappings to support VegX's concept of pointers between objects. Aaron Marcuse-Kubitza
05:22 PM Revision 34: data2xml: Small correction to NYBG mapping
Aaron Marcuse-Kubitza
04:58 PM Revision 33: data2xml: Created simplify_xpath script to remove duplication from XPath expressions
Aaron Marcuse-Kubitza
04:15 PM Revision 32: data2xml: Added support for * abbrs for backward (child-to-parent) pointers
Aaron Marcuse-Kubitza
02:52 PM Revision 31: In data2xml, fixed determination of which nesting level to put IDs on
Aaron Marcuse-Kubitza
02:45 PM Revision 30: Simplified expansion of * abbrs
Aaron Marcuse-Kubitza
02:23 PM Revision 29: Removed no longer necessary strip() from node value getter
Aaron Marcuse-Kubitza
02:22 PM Revision 28: Added patch for xml.dom.minidom.Element.writexml to avoid adding extra whitespace around text nodes
Aaron Marcuse-Kubitza
12:45 PM Revision 27: Added pointer field name abbreviations to data2xml and NYBG mappings
Aaron Marcuse-Kubitza

11/22/2011

04:35 PM Revision 26: In data2xml, fixed pointer handling to deal with pointer targets that are themselves pointers
Aaron Marcuse-Kubitza
04:01 PM Revision 25: In data2xml, added shortcut for lookahead assertion using ! symbol
Aaron Marcuse-Kubitza
02:32 PM Revision 24: In data2xml, fixed backward (child-to-parent) pointer handling to get and set attribute values properly
Aaron Marcuse-Kubitza
01:52 PM Revision 23: In data2xml, fixed xpath.get() to do last_only optimization properly for pointer targets
Aaron Marcuse-Kubitza
01:32 PM Revision 22: In data2xml, added support for XPath pointers
Aaron Marcuse-Kubitza

11/21/2011

05:48 PM Revision 21: Merged data2xml XPath functionality into xpath.py. Merged data2xml xml_dom.py and xml2db xml_util.py into identical xml_util.py for each script.
Aaron Marcuse-Kubitza
04:50 PM Revision 20: Added empty_db script to reset the vegbank database after running xml2db/test in commit mode
Aaron Marcuse-Kubitza
02:20 PM Task #287: XML to database conversion script (merged into CSV-XML-database mapping script)
The script is on nimoy at @/home/bien_shared/svn/scripts/xml2db@. You can run the demo at @/home/bien_shared/svn/scri... Aaron Marcuse-Kubitza
02:01 PM Revision 19: Changed xml2db and vegbank db to be owned by new user vegbank
Aaron Marcuse-Kubitza

11/18/2011

05:38 PM Revision 18: Changed xml2db and data2xml to help standardize mapping to different XML formats
Aaron Marcuse-Kubitza
02:48 PM Revision 17: Added DROP DATABASE and CREATE DATABASE to vegbank.sql
Aaron Marcuse-Kubitza
12:52 PM Revision 16: Changed xml2db to use primarily node contents to determine whether a node is a field or a child table
Aaron Marcuse-Kubitza

11/17/2011

04:42 PM Revision 15: Changed xml2db to use the first column in a table as its primary key
Aaron Marcuse-Kubitza
04:08 PM Revision 14: Changed xml2db to avoid inserting duplicate rows
Aaron Marcuse-Kubitza
03:31 PM Revision 13: Initial version of xml2db. Doesn't yet handle all duplicate rows correctly.
Aaron Marcuse-Kubitza
11:29 AM Revision 12: Removed .pyc files
Aaron Marcuse-Kubitza
10:29 AM Revision 11: Added BIEN 3 scripts
Aaron Marcuse-Kubitza
10:08 AM Task #287 (Resolved): XML to database conversion script (merged into CSV-XML-database mapping script)
Python script to import an XML file into a PostgreSQL database
Aaron Marcuse-Kubitza
10:07 AM Task #286 (New): CSV-XML-database mapping script
Python script to map CSV, XML, and database datasources to each other, using a map spreadsheet when needed Aaron Marcuse-Kubitza
10:06 AM Task #285 (Resolved): CSV to XML mappings for NYBG, SALVIAS
CSV spreadsheets mapping CSV file columns to XPath expressions for the NYBG and SALVIAS datasources Aaron Marcuse-Kubitza
 

Also available in: Atom