7bea00a401
Full v2 scorer benchmark data across 29 models (20 base + 9 LEK-tuned): - P20 (21 probes): All 29 models, 3 conditions each - P100 (101 probes): Top 5 models + LEK-4B, publication-quality data Key findings: - LEK-1B (21.74) beats base 4B/12B/27B at P100 scale — no kernel needed - Emergent realignment resistance: LEK models degrade with runtime kernel - Gemma3-12B + JSON kernel = 23.66 (best kernel-boosted score) - Family lineages: Mistral 3.80→14.58, Qwen regressed then recovered New scripts: ab_test.py (v2 scorer), self_distill.py (curriculum generation), extract_training.py, rephrase_probes.py, Phase 0/1 runners New seeds: P01-P100 merged (101 probes), 404 rephrased variants, 50 creative prompts for Phase 0 baseline lock 27B curriculum design: 4-phase staged training targeting 25+ baseline Co-Authored-By: Virgil <virgil@lethean.io>
55 lines
1.8 KiB
Bash
Executable file
55 lines
1.8 KiB
Bash
Executable file
#!/bin/bash
|
||
# Phase 0: Baseline Lock — Creative writing data generation
|
||
#
|
||
# Run Gemma3-27B (NO kernel) on creative prompts
|
||
# Generate 10 samples each at temperature 0.9 (more creative variance)
|
||
# No v2 threshold — creative quality needs manual review, not axiom scoring
|
||
# This protects creative capability from being lost in later phases
|
||
#
|
||
# Expected: ~50 prompts × 10 samples × ~45s = ~6 hours
|
||
# Produces: raw creative outputs for manual curation
|
||
|
||
SCRIPTS="/Volumes/Data/lem/scripts"
|
||
MODEL="/Volumes/Data/lem/gemma-3-27b-it-base"
|
||
PROBES="/Volumes/Data/lem/seeds/phase0-creative.json"
|
||
TRAIN_DIR="/Volumes/Data/lem/training"
|
||
|
||
mkdir -p "$TRAIN_DIR"
|
||
|
||
echo "=== Phase 0: Creative Baseline Lock ==="
|
||
echo "Model: $MODEL"
|
||
echo "Probes: $PROBES (creative, no axiom content)"
|
||
echo "Kernel: NONE (pure creative, no ethics kernel)"
|
||
echo "Threshold: 15.0 (structural only — keeps anything coherent)"
|
||
echo "Temperature: 0.9 (higher creative variance)"
|
||
echo "Samples: 10 per prompt"
|
||
echo ""
|
||
|
||
# Step 1: Generate creative data (no kernel — baseline creativity)
|
||
echo "--- Step 1: Creative generation ---"
|
||
python3 "$SCRIPTS/self_distill.py" \
|
||
--model "$MODEL" \
|
||
--prompts "$PROBES" \
|
||
--output "$TRAIN_DIR/phase0-raw.jsonl" \
|
||
--samples 10 \
|
||
--threshold 15.0 \
|
||
--max-tokens 4096 \
|
||
--temperature 0.9
|
||
|
||
echo ""
|
||
|
||
# Step 2: Extract all passing samples
|
||
echo "--- Step 2: Extract creative data ---"
|
||
python3 "$SCRIPTS/extract_training.py" \
|
||
--input "$TRAIN_DIR/phase0-raw.jsonl" \
|
||
--output "$TRAIN_DIR/phase0-train-all.jsonl" \
|
||
--dedup all \
|
||
--stats
|
||
|
||
echo ""
|
||
echo "=== Phase 0 data generation complete ==="
|
||
echo "Raw: $TRAIN_DIR/phase0-raw.jsonl"
|
||
echo "Training: $TRAIN_DIR/phase0-train-all.jsonl"
|
||
echo ""
|
||
echo "NEXT: Manual review of creative quality."
|
||
echo "Phase 0 trains BEFORE Phase 1 — protects creative regression."
|