Project

General

Profile

Statistics
| Revision:

# Date Author Comment
6290 11/19/2012 03:45 PM Aaron Marcuse-Kubitza

inputs/Makefile: Import logs: $(rsyncLogs): Include log files at any depth in the directory tree rather than just 1-2 levels deep. This adds log files whose containing directories have been moved to _archive/ directories.

6289 11/19/2012 03:29 PM Aaron Marcuse-Kubitza

Added inputs/_archive/

6288 11/19/2012 03:27 PM Aaron Marcuse-Kubitza

Removed inputs/QMOR/ because it's an insect collection

6287 11/19/2012 03:25 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: projectcontributor: Removed surname, since this information is stored in party_id->party.surname

6286 11/19/2012 03:23 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: projectcontributor: Removed cheatrole, since there is already a role field and this field was unused in VegBank

6285 11/19/2012 03:21 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: role: Added values from projectcontributor.ROLE_ID <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=projectcontributor&entity=dba_tabledescription&where=where_tablename#ROLE_ID>

6284 11/19/2012 03:17 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: sourcecontributor: role: Changed type to role

6283 11/19/2012 03:15 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: role enum: Added VegBank data dictionary values from <http://vegbank.org/vegbank/views/dba_fielddescription_detail.jsp?view=detail&wparam=1331&entity=dba_fielddescription&params=1331>

6282 11/19/2012 03:03 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: sourcecontributor: Renamed position to order for consistency with the ERD definition <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=referencecontributor&entity=dba_tabledescription&where=where_tablename#position> and disambiguation from other meanings of position which are similar to role

6281 11/19/2012 03:00 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: sourcecontributor: Renamed roletype to role for consistency with the ERD definition <http://vegbank.org/vegbank/views/dba_tabledescription_detail.jsp?view=detail&wparam=referencecontributor&entity=dba_tabledescription&where=where_tablename#roleType>

6280 11/19/2012 02:53 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/map.csv: Mapped to county, acceptedCounty

6279 11/19/2012 02:52 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped acceptedCounty, county to the matched place

6278 11/19/2012 02:50 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Added acceptedCounty

6277 11/19/2012 02:42 PM Aaron Marcuse-Kubitza

schemas/pg_hba.Mac.conf: Changed to match schemas/pg_hba.conf

6276 11/19/2012 02:37 PM Aaron Marcuse-Kubitza

schemas/pg_hba.conf: Fixed bug where also need an IPv6 bien entry with md5 authentication, because the IPv4 md5 authentication does not apply to "localhost", which is translated to the IPv6 address ::1

6275 11/19/2012 02:27 PM Aaron Marcuse-Kubitza

schemas/pg_hba.conf: Fixed bug where also need a local bien entry with md5 authentication, because the host-based md5 authentication applies only to literal IP addresses, not "localhost"

6274 11/19/2012 02:08 PM Aaron Marcuse-Kubitza

Added schemas/pg_hba.Mac.conf

6273 11/19/2012 02:01 PM Aaron Marcuse-Kubitza

schemas/pg_hba.conf: Restrict all accesses to the server except the bien user accessing vegbien using ident or a password, and the postgres superuser logging in using ident

6272 11/19/2012 01:25 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_output/map.csv: Mapped countyvalidity to latLongInCounty

6271 11/19/2012 01:24 PM Aaron Marcuse-Kubitza

schemas/functions.sql: _map(): Fixed bug where entries that map to NULL were incorrectly being treated as if the entry didn't exist. Note that -> returns NULL both if the entry's value is NULL and if the entry doesn't exist, so ? must be used to recheck the presence of the key in the hstore.

6270 11/19/2012 12:48 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped latLongInCounty

6269 11/19/2012 12:46 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Added latLongInCounty

6268 11/19/2012 12:43 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Added distance_to_county_m. Note that this can also be used to store latLongInCounty by mapping true to 0 and false to -1.

6267 11/19/2012 12:22 PM Aaron Marcuse-Kubitza

schemas/pg_hba.conf: Changed trust authentication back to ident/md5. Not sure how it got set to trust since I used md5 when enabling remote access to the DB for the bien user.

6266 11/19/2012 12:08 PM Aaron Marcuse-Kubitza

Added schemas/pg_hba.conf

6265 11/19/2012 11:48 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: place: Removed placecode to prevent datasources from creating duplicate entries for the same place, with different placecodes. This was a problem with the original BIEN2 geoscrub dataset, which contained duplicates.

6264 11/19/2012 10:54 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

6263 11/16/2012 07:06 PM Aaron Marcuse-Kubitza

Regenerated vegbien.ERD exports

6262 11/16/2012 07:03 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: Fixed bug where need to join taxonoccurrence.collector_id to party because it's now an fkey rather than a literal name

6261 11/16/2012 06:58 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_*: Added coordinateUncertaintyInMeters

6260 11/16/2012 06:34 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: analytical_stem_view: Join to newWorldCountries on 2-digit ISO code instead of country name, to increase (BIEN2) newWorldCountries and GADM overlap

6259 11/16/2012 06:29 PM Aaron Marcuse-Kubitza

psql_vegbien: Run with sh because it no longer needs bash support

6258 11/16/2012 06:28 PM Aaron Marcuse-Kubitza

psql_script_vegbien: Fixed bug where needs to be run with bash instead of sh

6257 11/16/2012 06:27 PM Aaron Marcuse-Kubitza

Added inputs/newWorld/iso_code_gadm/

6256 11/16/2012 06:16 PM Aaron Marcuse-Kubitza

Added inputs/newWorld/_src/

6255 11/16/2012 06:15 PM Aaron Marcuse-Kubitza

inputs/XAL/Specimen/map.csv: darwin:FieldNumber: Removed command to determine that field is unused, because UNUSED is a factual assertion that does not need a reason to be specified each time

6254 11/16/2012 06:11 PM Aaron Marcuse-Kubitza

inputs/XAL/Specimen/map.csv: Remapped darwin:CoordinatePrecision to UNUSED

6253 11/16/2012 06:08 PM Aaron Marcuse-Kubitza

inputs/NY/Specimen/map.csv: Remapped CoordinatePrecision to coordinateUncertaintyInMeters, assuming units of m based on the range and precision of values

6252 11/16/2012 06:03 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: coordinatePrecision: Added units (degrees) to form coordinatePrecision_deg

6251 11/16/2012 06:00 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Removed mapping for coordinatePrecision, which is not the same as coordsaccuracy_m. coordinatePrecision is instead "the precision of the coordinates" themselves in degrees (<http://rs.tdwg.org/dwc/terms/#coordinatePrecision&gt;).

6250 11/16/2012 05:53 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: coordinates: Changed coordinates.coordsaccuracy_deg units to m

6249 11/16/2012 05:51 PM Aaron Marcuse-Kubitza

Regenerated inputs/bien_web/observation/test.xml.ref

6248 11/16/2012 05:17 PM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: Added projectcontributor, locationeventcontributor to ERD

6247 11/16/2012 05:02 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: higher_plant_group_nodes: Added root->NULL mapping to store all the families that don't match any higher plant group

6246 11/16/2012 04:58 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: higher_plant_group_nodes: Allow NULL values for higher_plant_group, to allow mapping all remaining families to NULL in family_higher_plant_group

6245 11/16/2012 04:09 PM Aaron Marcuse-Kubitza

psql_vegbien: Fixed bug where did not display command prompt when run from command line, by moving automatic setting of search_path to psql_script_vegbien. psql_script_vegbien is now used instead of psql_vegbien wherever the search_path needs to be set, so removing this functionality from psql_vegbien is not a problem.

6244 11/16/2012 04:03 PM Aaron Marcuse-Kubitza

input.Makefile: BIEN commands: $(psqlAsBien): Use psql_script_vegbien, which automatically adds the $(psqlOpts), instead of psql_vegbien

6243 11/16/2012 03:54 PM Aaron Marcuse-Kubitza

schemas/functions.sql: _map(): Support any entry having the value '*' (not just the '*' entry), which passes through that value. Support an entry having the value '!', which raises an exception.

6242 11/16/2012 03:40 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/plotMetadata_/map.csv: AccessCode: Removed _map entry for 4, which does not apply to plots

6241 11/16/2012 03:07 PM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: Fixed lines

6240 11/16/2012 01:00 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: locationevent: Added accesslevel

6239 11/16/2012 12:54 PM Aaron Marcuse-Kubitza

inputs/SALVIAS/plotMetadata_/map.csv: Mapped AccessCode to dcterms:accessRights with appropriate _map filter

6238 11/16/2012 12:49 PM Aaron Marcuse-Kubitza

Added inputs/.geoscrub/geoscrub_cleaned_unique/_no_import to disable geoscrub_cleaned_unique, since the new geoscrub_output supersedes it

6237 11/16/2012 12:47 PM Aaron Marcuse-Kubitza

Added inputs/.geoscrub/geoscrub_output/

6236 11/16/2012 12:46 PM Aaron Marcuse-Kubitza

Added inputs/.geoscrub/_src/README.TXT

6235 11/16/2012 12:29 PM Aaron Marcuse-Kubitza

Regenerated inputs/bien_web/observation/VegBIEN.csv

6234 11/16/2012 12:24 PM Aaron Marcuse-Kubitza

Added inputs/.geoscrub/_src/ to store Jim's geoscrub CSV

6233 11/16/2012 12:21 PM Aaron Marcuse-Kubitza

schemas/functions.sql: _map(): Changed error message for an unmapped value to "Value not in map" rather than "Invalid map value", because an unmapped value is not necessarily explicitly invalid

6232 11/16/2012 12:16 PM Aaron Marcuse-Kubitza

inputs/VegBank/plot_/map.csv: confidentialitystatus filter: Merged mappings for 0 with other public-equivalent fields. Note that fuzzed plots are still public, because the private columns have been removed.

6231 11/15/2012 11:16 PM Aaron Marcuse-Kubitza

inputs/VegBank/plot_/map.csv: Mapped confidentialitystatus to dcterms:accessRights with an appropriate _map filter

6230 11/15/2012 11:16 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped dcterms:accessRights

6229 11/15/2012 11:14 PM Aaron Marcuse-Kubitza

schemas/functions.sql: _map(): Raise data_exception if value not in map and no default provided (not the same as a NULL default value)

6228 11/15/2012 10:54 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: verbatimGrowthForm: Removed _map filter, which applied only to SALVIAS and has now been moved to the applicable SALVIAS tables

6227 11/15/2012 10:51 PM Aaron Marcuse-Kubitza

inputs/SALVIAS*/plotObservations/map.csv: Remapped Habit to growthForm with _map filter applied

6226 11/15/2012 10:43 PM Aaron Marcuse-Kubitza

sql_io.py: put_table(): Special handling for functions with hstore params: Fixed bug where need to unwrap literal values of mapping, which might be sql_gen.Literal objects

6225 11/15/2012 10:43 PM Aaron Marcuse-Kubitza

sql_gen.py: Added get_value()

6224 11/15/2012 10:42 PM Aaron Marcuse-Kubitza

dicts.py: join(): Added support for unhashable types, which are passed through. This adds support for SQL literal values which are dicts (hstores).

6223 11/15/2012 10:25 PM Aaron Marcuse-Kubitza

xml_func.py: Removed no longer used _map(), which has been replaced by a corresponding DB function

6222 11/15/2012 10:22 PM Aaron Marcuse-Kubitza

schemas/functions.sql: Added _map(), which uses the new hstore functionality. This expands _map() functionality to column-based import.

6221 11/15/2012 10:20 PM Aaron Marcuse-Kubitza

root Makefile: VegBIEN DB: DB and bien user: mk_db: hstore extension: Fixed bug where need to use `CREATE EXTENSION hstore SCHEMA pg_catalog` instead of createlang, because hstore must be explicitly created in pg_catalog or else it will be created in the public schema instead, causing it to get deleted every time the public schema is reinstalled and cascading the delete to everything (including in other schemas) that uses hstore

6220 11/15/2012 10:04 PM Aaron Marcuse-Kubitza

sql_io.py: put_table(): Added special handling for functions with hstore params. Note that although _map() doesn't exist yet as a DB function, this code must be in place before _map() is created to avoid param type mismatch errors.

6219 11/15/2012 08:57 PM Aaron Marcuse-Kubitza

root Makefile: PostgreSQL: postgres-Linux: Changed plpython to plpython3 in order to install plpython3u

6218 11/15/2012 08:30 PM Aaron Marcuse-Kubitza

schemas/py_functions.sql: _date(): Removed features that require dateutil, which is not available under plpython3u. This includes removing the now-unused date string parameter.

6217 11/15/2012 08:26 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Removed _date/date, because _date using a string date argument is no longer supported under plpython3u (dateutil is missing). Note that PostgreSQL's own date parsing is sufficient for most dates, so this use of _date is not strictly necessary and removing it will improve import times.

6216 11/15/2012 08:12 PM Aaron Marcuse-Kubitza

schemas/py_functions.sql: Replaced xrange() with range() for plpython3u

6215 11/15/2012 08:05 PM Aaron Marcuse-Kubitza

root Makefile: Python: python-Linux: Also install python3, needed by plpython3u

6214 11/15/2012 08:04 PM Aaron Marcuse-Kubitza

schemas/py_functions.sql: Updated except clause syntax for PostgreSQL 9.1.6

6213 11/15/2012 08:03 PM Aaron Marcuse-Kubitza

schemas/*.sql: Updated for PostgreSQL 9.1.6, which has standard_conforming_strings = on (which affects \-escapes in string literals), escape_string_warning not explicitly set, and uses ALTER TABLE ONLY instead of ALTER TABLE

6212 11/15/2012 07:49 PM Aaron Marcuse-Kubitza

README.TXT: Removed step to manually run make_analytical_db, now that this is done automatically by import_all. Added separate instructions to remake the analytical DB.

6211 11/15/2012 07:45 PM Aaron Marcuse-Kubitza

import_all: Change to main directory make targets are run from. Use relative paths to bin/ commands, which is possible now that the current dir is set.

6210 11/15/2012 07:41 PM Aaron Marcuse-Kubitza

import_all: Create a background process that waits until the import is done and then runs make_analytical_db

6209 11/15/2012 07:36 PM Aaron Marcuse-Kubitza

Added waitpid

6208 11/15/2012 06:52 PM Aaron Marcuse-Kubitza

import_all: Documented that `wait %1` waits for asynchronous commands

6207 11/15/2012 06:40 PM Aaron Marcuse-Kubitza

root Makefile: VegBIEN DB: DB and bien user: mk_db: Also install hstore extension. Note that this is only supported by PostgreSQL 9.1+.

6206 11/15/2012 06:33 PM Aaron Marcuse-Kubitza

input.Makefile: Editing import: Updated queries for current schema

6205 11/15/2012 06:27 PM Aaron Marcuse-Kubitza

inputs/.geoscrub/geoscrub_cultivated/create.sql: Fixed bug where need to filter out NULL lat/longs because primary keys can't contain NULL values

6204 11/15/2012 06:17 PM Aaron Marcuse-Kubitza

schemas/py_functions.sql: Changed function languages to plpython3u to match the new installed version. Note that plpythonu is not available on Mac under PostgreSQL 9.1.6.

6203 11/15/2012 05:59 PM Aaron Marcuse-Kubitza

reinstall_all: Fixed bug where also need to include datasources starting with . such as .TNRS/, by using with_all's new $all option

6202 11/15/2012 05:58 PM Aaron Marcuse-Kubitza

with_all: Added $all option to also include datasources starting with . such as .TNRS/. This is necessary for reinstall_all, which needs to install all datasources.

6201 11/15/2012 05:18 PM Aaron Marcuse-Kubitza

root Makefile: PostgreSQL: $(pg_ctl-*): Fixed bug where need to pause for a few seconds after restarting PostgreSQL, to wait for the server to be ready to accept connections

6200 11/15/2012 05:12 PM Aaron Marcuse-Kubitza

root Makefile: Installation: uninstall: Removed inputs/uninstall because the DB will be uninstalled anyway, so the inputs don't need to be individually removed first

6199 11/15/2012 05:11 PM Aaron Marcuse-Kubitza

schemas/postgresql.Mac.conf: Added back unix_socket_directory setting, which is apparently still needed in PostgreSQL 9.1.6

6198 11/15/2012 05:06 PM Aaron Marcuse-Kubitza

root Makefile: PostgreSQL: postgres-Linux: Also install postgresql.conf

6197 11/15/2012 04:54 PM Aaron Marcuse-Kubitza

root Makefile: PostgreSQL: postgres-Darwin: Also install postgresql.Mac.conf

6196 11/15/2012 04:40 PM Aaron Marcuse-Kubitza

root Makefile: PostgreSQL: $(macUsePostgresLib): Factored out PostgreSQL dir to $(macPostgresDir)

6195 11/15/2012 04:38 PM Aaron Marcuse-Kubitza

schemas/postgresql.Mac.conf: Updated to PostgreSQL 9.1.6's postgresql.conf

6194 11/15/2012 04:29 PM Aaron Marcuse-Kubitza

root Makefile: Datasources: inputs/install: Fixed bug where need to `wait` after `. bin/reinstall_all` to wait for inputs to finish installing before installing the public schema. This is necessary because views in the public schema now have dependencies on some datasources, such as TNRS.

6193 11/15/2012 04:25 PM Aaron Marcuse-Kubitza

root Makefile: PostgreSQL: $(psqlAsAdmin): Use new $(asAdmin)

6192 11/15/2012 04:25 PM Aaron Marcuse-Kubitza

root Makefile: VegBIEN DB: Schemas: schemas/public/install: Use $(psqlNoSearchPath) instead of $(psqlAsBien) because the search_path is set by vegbien.sql

6191 11/15/2012 04:16 PM Aaron Marcuse-Kubitza

root Makefile: Datasources: Added inputs/install override which runs `. bin/reinstall_all` instead, in order to install all datasources simultaneously