dataset: bank

goal: What's the subscription rate for month='may' AND job='management' AND balance in Q1?

expected_output_type: scalar

answer_type: scalar

verification_code:
df['_q'] = pd.qcut(df['balance'], 4, labels=['Q1','Q2','Q3','Q4'], duplicates='drop')
filtered = df[(df['month'] == 'may') & (df['job'] == 'management') & (df['_q'] == 'Q1')]
result = round((filtered['y'] == 1).mean() * 100, 2) if len(filtered) > 0 else 0.0
df.drop('_q', axis=1, inplace=True)

ambiguities: quartile how define q1 q2 q3 q4 bins bucket positive success target y=1 class outcome null missing nan empty na

success_criteria:
Answer must match expected value
Numeric tolerance: 0.01

constraints: Complex analysis with 3+ operations expected

col_a: month

val_a: may

col_b: job

val_b: management

metric: balance

target_col: y

target_desc: subscription


goal: Among customers in top 95% of age AND bottom 10% of duration, what's subscription rate?

verification_code:
p_high = df['age'].quantile(95 / 100)
p_low = df['duration'].quantile(10 / 100)
cohort = df[(df['age'] >= p_high) & (df['duration'] <= p_low)]
result = round((cohort['y'] == 1).mean() * 100, 2) if len(cohort) > 0 else 0.0

ambiguities: percentile inclusive exclusive boundary include positive success target y=1 class outcome

metric_a: age

metric_b: duration


goal: Find the job with lowest average balance. What is the subscription rate for that segment?

verification_code:
group_stats = df.groupby('job')['balance'].mean()
extrema_val = group_stats.max() if 'lowest' == 'highest' else group_stats.min()
tied = group_stats[group_stats == extrema_val].sort_index()
extrema_group = tied.index[0]
subset = df[df['job'] == extrema_group]
result = round((subset['y'] == 1).mean() * 100, 2)

ambiguities: positive success target y=1 class outcome rate percentage decimal format 0-100 0-1

constraints: Multi-step analysis with 2 operations expected

group_col: job

extrema: lowest


goal: What's the subscription rate for month='may' AND job='management' AND age in Q4?

verification_code:
df['_q'] = pd.qcut(df['age'], 4, labels=['Q1','Q2','Q3','Q4'], duplicates='drop')
filtered = df[(df['month'] == 'may') & (df['job'] == 'management') & (df['_q'] == 'Q4')]
result = round((filtered['y'] == 1).mean() * 100, 2) if len(filtered) > 0 else 0.0
df.drop('_q', axis=1, inplace=True)

metric: age


goal: For month with above-average day, which have subscription rate > 25%? Return sorted list.

expected_output_type: list

answer_type: list

verification_code:
avg_metric = df.groupby('month')['day'].mean()
high_metric = avg_metric[avg_metric > avg_metric.mean()].index
rates = df[df['month'].isin(high_metric)].groupby('month')['y'].apply(
    lambda x: (x == 1).mean() * 100)
result = sorted(rates[rates > 25].index.tolist())

ambiguities: tie ties equal same duplicate

success_criteria: List elements must match (order matters)

group_col: month

metric: day


goal: Find the job with highest average balance. What is the subscription rate for that segment?

verification_code:
group_stats = df.groupby('job')['balance'].mean()
extrema_val = group_stats.max() if 'highest' == 'highest' else group_stats.min()
tied = group_stats[group_stats == extrema_val].sort_index()
extrema_group = tied.index[0]
subset = df[df['job'] == extrema_group]
result = round((subset['y'] == 1).mean() * 100, 2)

extrema: highest


goal: Among customers in top 90% of balance AND bottom 10% of day, what's subscription rate?

verification_code:
p_high = df['balance'].quantile(90 / 100)
p_low = df['day'].quantile(10 / 100)
cohort = df[(df['balance'] >= p_high) & (df['day'] <= p_low)]
result = round((cohort['y'] == 1).mean() * 100, 2) if len(cohort) > 0 else 0.0

metric_a: balance

metric_b: day


goal: For each job, compute volatility score: std(age) / mean(age). Return top 5 with [group, mean, std, volatility].

expected_output_type: dataframe

job: unemployed

job: self-employed

job: admin.

job: services

job: management

answer_type: dataframe

verification_code:
stats = df.groupby('job')['age'].agg(['mean', 'std']).round(2)
stats['volatility'] = round(stats['std'] / stats['mean'], 4)
result = stats.nlargest(5, 'volatility').reset_index()

ambiguities: round decimal precision digits volatility cv coefficient variation

success_criteria: Values must match with numeric tolerance 0.01


goal: Among customers in top 95% of balance AND bottom 10% of duration, what's subscription rate?

verification_code:
p_high = df['balance'].quantile(95 / 100)
p_low = df['duration'].quantile(10 / 100)
cohort = df[(df['balance'] >= p_high) & (df['duration'] <= p_low)]
result = round((cohort['y'] == 1).mean() * 100, 2) if len(cohort) > 0 else 0.0


goal: Among customers in top 90% of balance AND bottom 25% of duration, what's subscription rate?

verification_code:
p_high = df['balance'].quantile(90 / 100)
p_low = df['duration'].quantile(25 / 100)
cohort = df[(df['balance'] >= p_high) & (df['duration'] <= p_low)]
result = round((cohort['y'] == 1).mean() * 100, 2) if len(cohort) > 0 else 0.0


goal: Rank job by subscription rate. Which bottom-3 have above-median age?

answer_value: blue-collar entrepreneur

verification_code:
stats = df.groupby('job').agg(
    rate=('y', lambda x: (x == 1).mean()),
    avg_metric=('age', 'mean'))
stats['rank'] = stats['rate'].rank()
bottom_3 = stats[stats['rank'] <= 3]
result = sorted(bottom_3[bottom_3['avg_metric'] > stats['avg_metric'].median()].index.tolist())

ambiguities: positive success target y=1 class outcome rank order sort ascending descending


goal: For each month, compute volatility score: std(day) / mean(day). Return top 5 with [group, mean, std, volatility].

month: feb

month: sep

month: mar

month: jun

month: dec

verification_code:
stats = df.groupby('month')['day'].agg(['mean', 'std']).round(2)
stats['volatility'] = round(stats['std'] / stats['mean'], 4)
result = stats.nlargest(5, 'volatility').reset_index()


goal: Show the average balance breakdown by job. Include count and mean balance for each category, sorted by mean descending.

job: retired

job: unknown

job: student

job: entrepreneur

job: housemaid

job: technician

job: blue-collar

verification_code:
breakdown = df.groupby('job').agg(
    count=('balance', 'size'),
    mean_balance=('balance', lambda x: round(x.mean(), 2))
).sort_values('mean_balance', ascending=False)
result = breakdown.reset_index()

ambiguities: round decimal precision digits

constraints: Single aggregation or binning operation expected


goal: Find the job with lowest average age. What is the subscription rate for that segment?

verification_code:
group_stats = df.groupby('job')['age'].mean()
extrema_val = group_stats.max() if 'lowest' == 'highest' else group_stats.min()
tied = group_stats[group_stats == extrema_val].sort_index()
extrema_group = tied.index[0]
subset = df[df['job'] == extrema_group]
result = round((subset['y'] == 1).mean() * 100, 2)


goal: Find the month with the lowest subscription rate. Within that group, what is the average day?

verification_code:
group_rates = df.groupby('month')['y'].apply(lambda x: (x == 1).mean())
outer_val = group_rates.max() if 'lowest' == 'highest' else group_rates.min()
outer_tied = group_rates[group_rates == outer_val].sort_index()
outer_group = outer_tied.index[0]
subset = df[df['month'] == outer_group]
result = round(subset['day'].mean(), 2)

ambiguities: tie ties equal same duplicate positive success target y=1 class outcome

extrema_outer: lowest

extrema_inner: highest


goal: Divide customers into 4 day quartiles. What is the subscription percentage (0-100) in the highest (top 25%) (Q4) quartile?

verification_code:
df['_bin'] = pd.qcut(df['day'], 4, labels=['Q1','Q2','Q3','Q4'], duplicates='drop')
bin_data = df[df['_bin'] == 'Q4']
result = round((bin_data['y'] == 1).mean() * 100, 2)
df.drop('_bin', axis=1, inplace=True)

ambiguities: quartile how define q1 q2 q3 q4 bins bucket positive success target y=1 class outcome rate percentage decimal format 0-100 0-1

bin_col: day

quartile_desc: highest (top 25%)


goal: Among customers in top 95% of balance AND bottom 25% of duration, what's subscription rate?

verification_code:
p_high = df['balance'].quantile(95 / 100)
p_low = df['duration'].quantile(25 / 100)
cohort = df[(df['balance'] >= p_high) & (df['duration'] <= p_low)]
result = round((cohort['y'] == 1).mean() * 100, 2) if len(cohort) > 0 else 0.0


goal: Which job categories would have the biggest impact if brought to average subscription rate? Return top 3 by potential gain (count * rate gap), sorted by impact.

answer_value: blue-collar services entrepreneur

verification_code:
overall_rate = (df['y'] == 1).mean()
group_stats = df.groupby('job').agg(
    rate=('y', lambda x: (x == 1).mean()),
    count=('y', 'size')
)
group_stats['gap'] = overall_rate - group_stats['rate']
group_stats['potential'] = group_stats['count'] * group_stats['gap']
top_potential = group_stats[group_stats['gap'] > 0].nlargest(3, 'potential')
result = top_potential.index.tolist()


goal: Find the month with the lowest subscription rate. Within that group, what is the average age?

verification_code:
group_rates = df.groupby('month')['y'].apply(lambda x: (x == 1).mean())
outer_val = group_rates.max() if 'lowest' == 'highest' else group_rates.min()
outer_tied = group_rates[group_rates == outer_val].sort_index()
outer_group = outer_tied.index[0]
subset = df[df['month'] == outer_group]
result = round(subset['age'].mean(), 2)

extrema_inner: lowest


dataset: road

goal: Which lighting categories have the highest total reported accidents? Show breakdown with [lighting, count, total_num_reported_accidents, avg_num_reported_accidents] sorted by total descending.

lighting: dim

lighting: daylight

lighting: night

verification_code:
breakdown = df.groupby('lighting').agg(
    count=('num_reported_accidents', 'size'),
    total_num_reported_accidents=('num_reported_accidents', 'sum'),
    avg_num_reported_accidents=('num_reported_accidents', lambda x: round(x.mean(), 2))
).sort_values('total_num_reported_accidents', ascending=False)
result = breakdown.reset_index()

group_col: lighting

target_desc: reported accidents


goal: Which weather categories have the highest average accident risk? Show breakdown with [weather, count, avg_accident_risk] sorted by average descending.

weather: foggy

weather: rainy

weather: clear

verification_code:
breakdown = df.groupby('weather').agg(
    count=('accident_risk', 'size'),
    avg_accident_risk=('accident_risk', lambda x: round(x.mean(), 4))
).sort_values('avg_accident_risk', ascending=False)
result = breakdown.reset_index()

group_col: weather

target_desc: accident risk


goal: Divide records into 4 speed_limit quartiles. What is the average accident risk in the lower-middle (25-50%) (Q2) quartile?

verification_code:
df['_bin'] = pd.qcut(df['speed_limit'], 4, labels=['Q1','Q2','Q3','Q4'], duplicates='drop')
bin_data = df[df['_bin'] == 'Q2']
result = round(bin_data['accident_risk'].mean(), 4)
df.drop('_bin', axis=1, inplace=True)

quartile_desc: lower-middle (25-50%)


goal: Divide records into 4 curvature quartiles. What is the average accident risk in the upper-middle (50-75%) (Q3) quartile?

verification_code:
df['_bin'] = pd.qcut(df['curvature'], 4, labels=['Q1','Q2','Q3','Q4'], duplicates='drop')
bin_data = df[df['_bin'] == 'Q3']
result = round(bin_data['accident_risk'].mean(), 4)
df.drop('_bin', axis=1, inplace=True)

bin_col: curvature

quartile_desc: upper-middle (50-75%)


goal: How much higher is the average accident risk for lighting='daylight' compared to 'night'? Return difference.

verification_code:
avg_a = df[df['lighting'] == 'daylight']['accident_risk'].mean()
avg_b = df[df['lighting'] == 'night']['accident_risk'].mean()
result = round(avg_a - avg_b, 4)

val_a: daylight

val_b: night


goal: How much higher is the average accident risk for road_type='rural' compared to 'urban'? Return difference.

verification_code:
avg_a = df[df['road_type'] == 'rural']['accident_risk'].mean()
avg_b = df[df['road_type'] == 'urban']['accident_risk'].mean()
result = round(avg_a - avg_b, 4)

val_a: rural

val_b: urban