public-projects/visrecall
1
0
Fork 0
Code Issues Pull requests Releases 1 Wiki Activity
visrecall/WebInterface/Front-end/generate-experiment-files/main.ipynb

51 KiB
Raw Blame History

Experimental set-up:

This code will generate experimental files that can either be independently hosted on a website and run with recruited participants, or via our MTurk iPython notebook be used for launching Amazon Mechanical Turk (MTurk) HITs.

An experiment is composed of different sets of images:

  • target images are the images you want to collect attention data on - those are images that you provide (in directory sourcedir below)
  • tutorial images are images that will be shown to participants at the beginning of the experiment to get them familiarized with the codecharts set-up (you can reuse the tutorial image we provide, or provide your own in directory tutorial_source_dir below)
    • hint: if your images are very different in content from the images in our set, you may want to train your participants on your own images, to avoid a context switch between the tutorial and main experiment
  • sentinel images are images interspersed throughout the experiment where participant attention is guided to a very specific point on the screen, used as validation/calibration images to ensure participants are actually moving their eyes and looking where they're supposed to; the code below will intersperse images from the sentinel_target_images directory we provide throughout your experimental sequence
    • sentinel images can be interspersed throughout both the tutorial and target images, or excluded from the tutorial (via add_sentinels_to_tutorial flag below); we recommend having sentinel images as part of the tutorial to familiarize participants with such images as well

The code below will populate the rootdir task directory with #num_subject_files subject files for you, where each subject file corresponds to an experiment you can run on a single participant. For each subject file, a set of #num_images_per_sf will be randomly sampled from the sourcedir image directory. A set of #num_sentinels_per_sf sentinel images will also be sampled from the sentinel_imdir image directory, and distributed throughout the experiment. A tutorial will be generated at the beginning of the experiment with #num_imgs_per_tutorial randomly sampled from the tutorial_source_dir image directory, along with an additional #num_sentinels_per_tutorial sentinel files distributed throughout the tutorial (if add_sentinels_to_tutorial flag is set to true).

In [1]:
import os
import string
import random
import json 
import matplotlib.pyplot as plt
import numpy as np
import base64 
import glob
In [4]:
sourcedir = '10' # replace this with your own directory of experiment images
filldir = '1' # the filler images for False alarm and Correct rejection
blurdir = '10_blur' # blurry images
In [5]:
# PARAMETERS for generating subject files
num_subject_files = 1     # number of subject files to generate (i.e., # of mturk assignments that will be put up)    
num_images_per_sf = 20    # number of target images per subject file 
num_imgs_per_tutorial = 0 # number of tutorial images per subject file
num_sentinels_per_sf = 0  # number of sentinel images to distribute throughout the experiment (excluding the tutorial)
add_sentinels_to_tutorial = False # whether to have sentinel images as part of the tutorial
num_sentinels_per_tutorial = 0  # number of sentinel images to distribute throughout the tutorial

Another bit of terminology and experimental logistics involves buckets which are a way to distribute experiment stimuli so that multiple experiments can be run in parallel (and participants can be reused for different subsets of images). If you have a lot of images that you want to collect data on, and for each participant you are sampling a set of only #num_images_per_sf, then you might have to generate a large num_subject_files in order to have enough data points per image. A way to speed up data collection is to split all the target images into #num_buckets disjoint buckets, and then to generate subject files per bucket. Given that subject files generated per bucket are guaranteed to have a disjoint set of images, the same participant can be run on multiple subject files from different buckets. MTurk HITs corresponding to different buckets can be launched all at once. In summary, in MTurk terms, you can generate as many HITs as num_buckets specified below, and as many assignments per HIT as num_subject_files.

The way the codecharts methodology works, a jittered grid of alphanumeric triplets appears after every image presentation (whether it is a target, sentinel, or tutorial image), since a participant will need to indicate where on the preceding image s/he looked, by reporting a triplet. To avoid generating an excessive number of codecharts (that bulks up all the subject files), we can reuse some codecharts across buckets. The way we do this is by pre-generating #ncodecharts codecharts, and then randomly sampling from these when generating the individual subject files.

In [6]:
# we pre-generate some codecharts and sentinel images so that we can reuse these across participants and buckets 
# and potentially not have to generate as many files; these can be set to any number, and the corresponding code
# will just sample as many images as need per subject file

ncodecharts = num_subject_files*num_images_per_sf # number of codecharts to generate; can be changed
sentinel_images_per_bucket = num_subject_files*num_sentinels_per_sf # can be changed

# set these parameters
num_buckets = 1      # number of disjoint sets of subject files to create (for running multiple parallel HITs)
start_bucket_at = 0  # you can use this and the next parameter to generate more buckets if running the code later
which_buckets = [0]  # a list of specific buckets e.g., [4,5,6] to generate experiment data for

rootdir = '../assets/task_data' # where all the experiment data will be stored
if not os.path.exists(rootdir):
    print('Creating directory %s'%(rootdir))
    os.makedirs(rootdir)

real_image_dir = os.path.join(rootdir,'real_images')              # target images, distributed by buckets
                                                                  # (shared across buckets)
sentinel_image_dir = os.path.join(rootdir,'sentinel_images')      # sentinel images, distributed by buckets
sentinel_CC_dir = os.path.join(rootdir,'sentinel_CC')             # codecharts corresponding to the sentinel images
                                                                  # (shared across buckets)
#sentinel_targetim_dir = os.path.join(rootdir, 'sentinel_target')  
real_blurred_dir = os.path.join(rootdir,'real_blurred')
real_filler_dir = os.path.join(rootdir,'real_filler')
In [7]:
# this cell creates an `all_images` directory, copies images from sourcedir, and pads them to the required dimensions

import create_padded_image_dir

all_image_dir = os.path.join(rootdir,'all_images')
if not os.path.exists(all_image_dir):
    print('Creating directory %s'%(all_image_dir))
    os.makedirs(all_image_dir)
    
allfiles = []
for ext in ('*.jpeg', '*.png', '*.jpg'):
    allfiles.extend(glob.glob(os.path.join(sourcedir, ext)))
print("%d files copied from %s to %s"%(len(allfiles),sourcedir,all_image_dir))
    
image_width,image_height = create_padded_image_dir.save_padded_images(all_image_dir,allfiles)

20 files copied from 10 to ../assets/task_data/all_images
Padding 20 image files to dimensions: [1734,1340]...
Done!
In [8]:
blurred_image_dir = os.path.join(rootdir,'blurred_images')

if not os.path.exists(blurred_image_dir):
    print('Creating directory %s'%(blurred_image_dir))
    os.makedirs(blurred_image_dir)
    
blurfiles = []
for ext in ('*.jpeg', '*.png', '*.jpg'):
    blurfiles.extend(glob.glob(os.path.join(blurdir, ext)))
print("%d files copied from %s to %s"%(len(blurfiles),blurdir,blurred_image_dir))
    
image_width,image_height = create_padded_image_dir.save_padded_images(blurred_image_dir,blurfiles)

20 files copied from 10_blur to ../assets/task_data/blurred_images
Padding 20 image files to dimensions: [1734,1340]...
Done!
In [9]:
filler_image_dir = os.path.join(rootdir,'filler_images')

if not os.path.exists(filler_image_dir):
    print('Creating directory %s'%(filler_image_dir))
    os.makedirs(filler_image_dir)
    
fillerfiles = []
for ext in ('*.jpeg', '*.png', '*.jpg'):
    fillerfiles.extend(glob.glob(os.path.join(filldir, ext)))
print("%d files copied from %s to %s"%(len(fillerfiles),filldir,filler_image_dir))
    
image_width,image_height = create_padded_image_dir.save_padded_images(filler_image_dir,fillerfiles)

20 files copied from 1 to ../assets/task_data/filler_images
Padding 20 image files to dimensions: [1786,1340]...
Done!
In [10]:
# this cell creates the requested number of buckets and distributes images from `all_image_dir` to the corresponding
# bucket directories inside `real_image_dir`

from distribute_image_files_by_buckets import distribute_images

distribute_images(all_image_dir,real_image_dir,num_buckets,start_bucket_at)

Distributing images across 1 buckets
Populating ../assets/task_data/real_images/bucket0 with 20 images
In [11]:
distribute_images(blurred_image_dir,real_blurred_dir,num_buckets,start_bucket_at)

Distributing images across 1 buckets
Populating ../assets/task_data/real_blurred/bucket0 with 20 images
In [12]:
distribute_images(filler_image_dir,real_filler_dir,num_buckets,start_bucket_at)

Distributing images across 1 buckets
Populating ../assets/task_data/real_filler/bucket0 with 20 images

We create sentinel images by taking a small object (one of a: fixation cross, red dot, or image of a face) and choosing a random location for it on a blank image (away from the image boundaries by at least border_padding pixels). The code below creates #sentinel_images_per_bucket such sentinel images in each bucket.

Now that all the previous cells have generated the requisite image, codechart, sentinel, and tutorial files, the following code will generate num_subject_files individual subject files by sampling from the appropriate image directories and creating an experimental sequence.

Only run this if you want new subject json

In [11]:
gt_answers = []
reco_answers = []
start_subjects_at = 0     # where to start creating subject files at (if had created other subject files previously)
#if os.path.exists(os.path.join(rootdir,'subject_files/bucket0')):
#    subjfiles = glob.glob(os.path.join(rootdir,'subject_files/bucket0/*.json'))
#    start_subjects_at = len(subjfiles)



## GENERATING SUBJECT FILES 
subjdir = os.path.join(rootdir,'subject_files')
if not os.path.exists(subjdir):
    os.makedirs(subjdir)
    #os.makedirs(os.path.join(rootdir,'full_subject_files'))


    
# iterate over all buckets 
for b in range(len(which_buckets)): 

    bucket = 'bucket%d'%(which_buckets[b])
    img_bucket_dir = os.path.join(real_image_dir,bucket)
    blur_bucket_dir = os.path.join(real_blurred_dir,bucket)
    filler_bucket_dir = os.path.join(real_filler_dir,bucket)
    img_files = []
    blur_files = []
    filler_files = []
    for ext in ('*.jpeg', '*.png', '*.jpg'):
        img_files.extend(glob.glob(os.path.join(img_bucket_dir, ext)))
    
    for ext in ('*.jpeg', '*.png', '*.jpg'):
        blur_files.extend(glob.glob(os.path.join(blur_bucket_dir, ext)))
    print(blur_files)
    for ext in ('*.jpeg', '*.png', '*.jpg'):
        filler_files.extend(glob.glob(os.path.join(filler_bucket_dir, ext)))
    
    filler_files.extend(img_files) #40 images for recognition task
    random.shuffle(filler_files)
    
    print(os.path.join(blur_bucket_dir, ext))
    #sentinel_bucket_dir = os.path.join(sentinel_image_dir,bucket)
    #sentinel_files = glob.glob(os.path.join(sentinel_bucket_dir,'*.jpg'))
    
    #with open(os.path.join(sentinel_bucket_dir,'sentinel_codes_full.json')) as f:
    #    sentinel_codes_data = json.load(f) # contains mapping of image path to valid codes
        
    subjdir = os.path.join(rootdir,'subject_files',bucket)
    if not os.path.exists(subjdir):
        os.makedirs(subjdir)
        #os.makedirs(os.path.join(rootdir,'full_subject_files',bucket))
    
    print('Generating %d subject files in bucket %d'%(num_subject_files,b))
    # for each bucket, generate subject files 
    for i in range(num_subject_files):
        
        #random.shuffle(img_files)
        #random.shuffle(sentinel_files)
        #random.shuffle(real_codecharts)

        # for each subject files, add real images 
        sf_data = []
        full_sf_data = []

        
        # initialize temporary arrays, because will shuffle real & sentinel tutorial images before adding to
        # final subject files
        sf_data_temp = []
        full_sf_data_temp = []
        
       
           
        
        # ADDING REAL IMAGES 
        for j in range(int(num_images_per_sf/2)):
            for k in range(2):
                image_data = {}
                image_data["image"] = img_files[j*2+k] # stores image path 

                # select a code chart
                #pathname = real_codecharts[j*2+k] # since shuffled, will pick up first set of random codecharts

                #image_data["codechart"] = pathname # stores codechart path 
                #image_data["codes"] = real_codes_data[pathname]['valid_codes'] # stores valid codes 
                image_data["flag"] = 'real' # stores flag of whether we have real or sentinel image

                full_image_data = image_data.copy() # identical to image_data but includes a key for coordinates
                #full_image_data["coordinates"] = real_codes_data[pathname]['coordinates'] # store locations - (x, y) coordinate of each triplet 

                sf_data.append(image_data)
                full_sf_data.append(full_image_data)
            
            for w in range(2):
                blur_data = {}
                print(blur_files)
                blur_data['image'] = blur_files[j*2+w]
                # on Windows comment this out
                #QA_file_name = blur_files[j*2+w].split('\\')[-1][:-4]+'.json'
                QA_file_name = blur_files[j*2+w].split('/')[-1][:-4]+'.json'
         
                with open(os.path.join(sourcedir,QA_file_name)) as f:
                    print(QA_file_name)
                    blur_data['QA'] = json.load(f)
                    #print(blur_data['QA'])
                    for item in blur_data['QA']:
                        if 'answer' in blur_data['QA'][item]:
                            gt_answers.append(blur_data['QA'][item]['answer'])
                #pathname = real_codecharts[j*2+w] # since shuffled, will pick up first set of random codecharts

                #blur_data["codechart"] = pathname # stores codechart path 
                #blur_data["codes"] = real_codes_data[pathname]['valid_codes'] # stores valid codes 
                blur_data["flag"] = 'blur'
                
                full_blur_data = blur_data.copy()
                #full_blur_data["coordinates"] = real_codes_data[pathname]['coordinates']
                
                sf_data.append(blur_data)
                full_sf_data.append(full_blur_data)
                
        for img in filler_files:
            filler_data = {}
            filler_data["image"] = img # stores image path 

                # select a code chart
            #pathname = real_codecharts[1] # since shuffled, will pick up first set of random codecharts

            #filler_data["codechart"] = pathname # stores codechart path 
            #filler_data["codes"] = real_codes_data[pathname]['valid_codes'] # stores valid codes 
            filler_data["flag"] = 'fill' # stores flag of whether we have real or sentinel image
            filler_data["showed"] = "real_images" in img
            if filler_data["showed"] == True:
                reco_answers.append('1')
            else:
                reco_answers.append('2')

            full_filler_data = filler_data.copy() # identical to image_data but includes a key for coordinates
            #full_filler_data["coordinates"] = real_codes_data[pathname]['coordinates'] # store locations - (x, y) coordinate of each triplet 

            sf_data.append(filler_data)
            full_sf_data.append(full_filler_data)
            
            

      

        # Add an image_id to each subject file entry
        image_id = 0 # represents the index of the image in the subject file 
        for d in range(len(sf_data)): 
            sf_data[d]['index'] = image_id
            full_sf_data[d]['index'] = image_id
            image_id+=1

        subj_num = start_subjects_at+i
        with open(os.path.join(rootdir,'subject_files',bucket,'subject_file_%d.json'%(subj_num)), 'w') as outfile: 
            print('Subject file %s DONE'%(outfile.name))
            json.dump(sf_data, outfile)
        with open(os.path.join(rootdir,'full_subject_files',bucket,'subject_file_%d.json'%(subj_num)), 'w') as outfile: 
            json.dump(full_sf_data, outfile)

['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
../assets/task_data/real_blurred/bucket0/*.jpg
Generating 1 subject files in bucket 0
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
wsj265.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
treasuryD07_3.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
economist_daily_chart_242.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
economist_daily_chart_257.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
economist_daily_chart_243.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
whoJ43_1.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
economist_daily_chart_150.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
3iRYXLvZ8oVQDMLR-CebnQ==.0.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
economist_daily_chart_194.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
whoQ12_2.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
whoF03.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
wsj3.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
whoB10_1.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
3LY3OX8bU7uKhgcRPgDRxw==.0.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
economist_daily_chart_262.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
economist_daily_chart_260.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
0BmWZbQdEukHi79Lit01oQ==.0.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
wsj86.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
whoJ36_2.json
['../assets/task_data/real_blurred/bucket0/wsj265.png', '../assets/task_data/real_blurred/bucket0/treasuryD07_3.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_242.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_257.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_243.png', '../assets/task_data/real_blurred/bucket0/whoJ43_1.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_150.png', '../assets/task_data/real_blurred/bucket0/3iRYXLvZ8oVQDMLR-CebnQ==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_194.png', '../assets/task_data/real_blurred/bucket0/whoQ12_2.png', '../assets/task_data/real_blurred/bucket0/whoF03.png', '../assets/task_data/real_blurred/bucket0/wsj3.png', '../assets/task_data/real_blurred/bucket0/whoB10_1.png', '../assets/task_data/real_blurred/bucket0/3LY3OX8bU7uKhgcRPgDRxw==.0.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_262.png', '../assets/task_data/real_blurred/bucket0/economist_daily_chart_260.png', '../assets/task_data/real_blurred/bucket0/0BmWZbQdEukHi79Lit01oQ==.0.png', '../assets/task_data/real_blurred/bucket0/wsj86.png', '../assets/task_data/real_blurred/bucket0/whoJ36_2.png', '../assets/task_data/real_blurred/bucket0/whoK21.png']
whoK21.json
Subject file ../assets/task_data/subject_files/bucket0/subject_file_0.json DONE
In [16]:
len(gt_answers)
Out[16]:
100
In [17]:
len(reco_answers)
Out[17]:
40
In [20]:
#import numpy as np
#np.save('gt_answers2',gt_answers)
#np.save('recogt_answers2',reco_answers)
In [ ]:
In [ ]: