HellaSwag
HellaSwag is a benchmark designed to evaluate language models' commonsense reasoning through sentence completion tasks. It provides 10,000 challenges spanning various subject areas. For more details, you can visit the Hellaswag GitHub page.
Hellaswag emphasizes commonsense reasoning and depth of understanding in real-world situations, making it an excellent tool for pinpointing where models might struggle with nuanced or complex contexts.
Arguments
There are two optional arguments when using the HellaSwag benchmark:
- [Optional]
tasks: a list of tasks (HellaSwagTaskenums), which specifies the subject areas for sentence completion evaluation. By default, this is set to all tasks. The list ofHellaSwagTaskenums can be found here. - [Optional]
n_shots: the number of "shots" to use for few-shot learning. This is set to 10 by default and cannot exceed 15.
Notice unlike BIGBenchHard, there is no CoT prompting for the HellaSwag benchmark.
Example
The code below evaluates a custom mistral_7b model (click here to learn how to use ANY custom LLM) and its ability to complete sentences related to 'Trimming Branchs or Hedges' and 'Baton Twirling' subjects using 5-shot learning.
from deepeval.benchmarks import HellaSwag
from deepeval.benchmarks.tasks import HellaSwagTask
# Define benchmark with specific tasks and shots
benchmark = HellaSwag(
tasks=[HellaSwagTask.TRIMMING_BRANCHES_OR_HEDGES, HellaSwagTask.BATON_TWIRLING],
n_shots=5
)
# Replace 'mistral_7b' with your own custom model
benchmark.evaluate(model=mistral_7b)
print(benchmark.overall_score)
The overall_score for this benchmark ranges from 0 to 1, where 1 signifies perfect performance and 0 indicates no correct answers. The model's score, based on exact matching, is calculated by determining the proportion of multiple-choice sentence-completion questions for which the model produces the precise correct letter answer (e.g. 'A') in relation to the total number of questions.
As a result, utilizing more few-shot prompts (n_shots) can greatly improve the model's robustness in generating answers in the exact correct format and boost the overall score.
HellaSwag Tasks
The HellaSwagTask enum classifies the diverse range of categories covered in the HellaSwag benchmark.
from deepeval.benchmarks.tasks import HellaSwagTask
hella_tasks = [HellaSwagTask.APPLYING_SUNSCREEN]
Below is the comprehensive list of available tasks:
APPLYING_SUNSCREENTRIMMING_BRANCHES_OR_HEDGESDISC_DOGWAKEBOARDINGSKATEBOARDINGWATERSKIINGWASHING_HANDSSAILINGPLAYING_CONGASBALLETROOF_SHINGLE_REMOVALHAND_CAR_WASHKITE_FLYINGPLAYING_POOLPLAYING_LACROSSELAYUP_DRILL_IN_BASKETBALLHOME_AND_GARDENPLAYING_BEACH_VOLLEYBALLCALF_ROPINGSCUBA_DIVINGMIXING_DRINKSPUTTING_ON_SHOESMAKING_A_LEMONADEUNCATEGORIZEDZUMBAPLAYING_BADMINTONPLAYING_BAGPIPESFOOD_AND_ENTERTAININGPERSONAL_CARE_AND_STYLECRICKETSHOVELING_SNOWPING_PONGHOLIDAYS_AND_TRADITIONSICE_FISHINGBEACH_SOCCERTABLE_SOCCERSWIMMINGBATON_TWIRLINGJAVELIN_THROWSHOT_PUTDOING_CRUNCHESPOLISHING_SHOESTRAVELUSING_UNEVEN_BARSPLAYING_HARMONICARELATIONSHIPSHIGH_JUMPMAKING_A_SANDWICHPOWERBOCKINGREMOVING_ICE_FROM_CARSHAVINGSHARPENING_KNIVESWELDINGUSING_PARALLEL_BARSHOME_CATEGORIESROCK_CLIMBINGSNOW_TUBINGWASHING_FACEASSEMBLING_BICYCLETENNIS_SERVE_WITH_BALL_BOUNCINGSHUFFLEBOARDDODGEBALLCAPOEIRAPAINTBALLDOING_A_POWERBOMBDOING_MOTOCROSSPLAYING_ICE_HOCKEYPHILOSOPHY_AND_RELIGIONARCHERYCARS_AND_OTHER_VEHICLESRUNNING_A_MARATHONTHROWING_DARTSPAINTING_FURNITUREHAVING_AN_ICE_CREAMSLACKLININGCAMEL_RIDEARM_WRESTLINGHULA_HOOPSURFINGPLAYING_PIANOGARGLING_MOUTHWASHPLAYING_ACCORDIONHORSEBACK_RIDINGPUTTING_IN_CONTACT_LENSESPLAYING_SAXOPHONEFUTSALLONG_JUMPLONGBOARDINGPOLE_VAULTBUILDING_SANDCASTLESPLATFORM_DIVINGPAINTINGSPINNINGCARVING_JACK_O_LANTERNSBRAIDING_HAIRYOUTHPLAYING_VIOLINCANOEINGCHEERLEADINGPETS_AND_ANIMALSKAYAKINGCLEANING_SHOESKNITTINGBAKING_COOKIESDOING_FENCINGPLAYING_GUITARRAUSING_THE_ROWING_MACHINEGETTING_A_HAIRCUTMOOPING_FLOORRIVER_TUBINGCLEANING_SINKGROOMING_DOGDISCUS_THROWCLEANING_WINDOWSFINANCE_AND_BUSINESSHANGING_WALLPAPERROPE_SKIPPINGWINDSURFINGKNEELINGGETTING_A_PIERCINGROCK_PAPER_SCISSORSSPORTS_AND_FITNESSBREAKDANCINGWALKING_THE_DOGPLAYING_DRUMSPLAYING_WATER_POLOBMXSMOKING_A_CIGARETTEBLOWING_LEAVESBULLFIGHTINGDRINKING_COFFEEBATHING_DOGTANGOWRAPPING_PRESENTSPLASTERINGPLAYING_BLACKJACKFUN_SLIDING_DOWNWORK_WORLDTRIPLE_JUMPTUMBLINGSKIINGDOING_KICKBOXINGBLOW_DRYING_HAIRDRUM_CORPSSMOKING_HOOKAHMOWING_THE_LAWNVOLLEYBALLLAYING_TILESTARTING_A_CAMPFIRESUMOHURLINGPLAYING_KICKBALLMAKING_A_CAKEFIXING_THE_ROOFPLAYING_POLOREMOVING_CURLERSELLIPTICAL_TRAINERHEALTHSPREAD_MULCHCHOPPING_WOODBRUSHING_TEETHUSING_THE_POMMEL_HORSESNATCHCLIPPING_CAT_CLAWSPUTTING_ON_MAKEUPHAND_WASHING_CLOTHESHITTING_A_PINATATAI_CHIGETTING_A_TATTOODRINKING_BEERSHAVING_LEGSDOING_KARATEPLAYING_RUBIK_CUBEFAMILY_LIFEROLLERBLADINGEDUCATION_AND_COMMUNICATIONSFIXING_BICYCLEBEER_PONGIRONING_CLOTHESCUTTING_THE_GRASSRAKING_LEAVESPLAYING_SQUASHHOPSCOTCHINSTALLING_CARPETPOLISHING_FURNITUREDECORATING_THE_CHRISTMAS_TREEPREPARING_SALADPREPARING_PASTAVACUUMING_FLOORCLEAN_AND_JERKCOMPUTERS_AND_ELECTRONICSCROQUET