Project

General

Profile

« Previous | Next » 

Revision 1650

Added by Aaron Marcuse-Kubitza about 12 years ago

inputs/REMIB/src/nodes.make: Abort prefix on empty response using break, rather than just done = True, to avoid running any more code except the finally block. Moved metadata row validation outside metadata row retrieval try-except block.

View differences:

nodes.make
67 67 
        node_url_template = url_template.replace('[node_id]', str(node_id))
68 68
69 69 
        for prefix_chars in itertools.product(alphabet, repeat=2):
70 
            if done: break
71 70 
            prefix = ''.join(prefix_chars)
72 71 
            log('Processing prefix '+prefix+'...')
73 72 
            row_ct = 0
......
79 78 
            stream = streams.StreamIter(streams.TimeoutInputStream(
80 79 
                urllib2.urlopen(url), timeout_))
81 80
82 
            # Copy lines
83 81 
            try:
84 82 
                util.skip(stream, is_ignore) # skip header
85 
                try:
86 
                    metadata_row = csv.reader(stream).next()
87 
                    if metadata_row[0] != 'COLLECTION': raise InputException(
88 
                        'Invalid metadata row: '+str(metadata_row))
83 
                try: metadata_row = csv.reader(stream).next()
89 84 
                except StopIteration:
90 85 
                    done = True # empty response means no more nodes
86 
                    break
87 
                if metadata_row[0] != 'COLLECTION': raise InputException(
88 
                    'Invalid metadata row: '+str(metadata_row))
91 89
90 
                # Copy lines
92 91 
                for line in stream:
93 92 
                    if is_ignore(line):
94 93 
                        error = strings.remove_prefix('\t\t', line)

Also available in: Unified diff