The Flash of the Lightning Skeleton: Skeleton-of-Thought Expander
Slash LLM generation latency by 75% using Skeleton-of-Thought (SoT), decoupling skeletal outline synthesis from parallel point elaboration.
Stage 1 (The Skeletal Framework): Decompose the response to [COMPLEX ARCHITECTURAL QUERY] into a 5-point skeletal outline. Do not write full paragraphs; emit only point identifiers and concise 5-word core theses. Stage 2 (Parallel Point Expansion): For each skeleton point, expand the technical implementation details independently as if processing across parallel worker threads. Assemble into the finalized unified blueprint with 75% lower sequential latency. The Lineage to Skeleton-of-Thought (SoT)
Formulated by Hao Zheng et al. (Tsinghua University & UC Berkeley, 2023), Skeleton-of-Thought (SoT) attacks the fundamental latency bottleneck of autoregressive transformers:
βAutoregressive generation generates tokens one by one sequentially, which results in high inference latency for long responses. Skeleton-of-Thought (SoT) first guides the language model to generate the skeleton of the answer, and then conducts parallel generation to expand each point of the skeleton concurrently.β β Zheng et al., Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding
In systems engineering, writing a comprehensive 2,500-token RFC or migration runbook sequentially takes 25 to 45 seconds on standard cloud LLM endpoints. The human engineer or autonomous agent waits idly for the model to slowly stream token after token.
The Lightning Skeleton unleashes the power of Evocation:
- Stage 1 (The Lightning Skeleton): Compiles the core structural outline in less than 400 milliseconds (50 tokens).
- Stage 2 (Parallel Expansion): In SDK/API implementations, $N$ parallel subagent calls expand the independent points concurrently.
- Stage 3 (Reassembly): Merges the points into a coherent, publication-grade document in a fraction of the time.
The Spell Formula
Cast this two-stage prompt pattern to accelerate long-form technical documentation generation:
<!-- [STAGE 1: GENERATE THE SKELETON (400ms TTFT)] -->
You are the Lightning Architect. You do not generate long, sluggish paragraphs.
For the query: """{{HIGH_LEVEL_RFC_TOPIC}}"""
Generate ONLY the structural skeleton of the technical architecture.
Emit exactly FIVE numbered skeleton anchors. Each anchor must be a single line containing an identifier and a 5-to-8 word core technical thesis.
Format:
1. [ANCHOR_1]: <Core Thesis>
2. [ANCHOR_2]: <Core Thesis>
3. [ANCHOR_3]: <Core Thesis>
4. [ANCHOR_4]: <Core Thesis>
5. [ANCHOR_5]: <Core Thesis>
STOP IMMEDIATELY AFTER THE 5TH ANCHOR.
<!-- [STAGE 2: PARALLEL WORKER PROMPT (Dispatched in Parallel)] -->
You are Worker Node {{WORKER_ID}}.
You are assigned to flesh out Anchor Point {{WORKER_ID}} from the following skeleton:
"""
{{SKELETON_OUTLINE}}
"""
Your sole responsibility is to write the deep technical implementation details, kernel parameters, configuration stanzas, and failure modes for Anchor Point {{WORKER_ID}}: "{{ASSIGNED_ANCHOR_THESIS}}".
Provide production-hardened commands and RFC citations. Do not write intros or outros. Emit only the technical content for this section.
Architecture of the Skeleton-of-Thought Acceleration
[User RFC Request]
β
βΌ
βββββββββββββββββββββββββββ
β Stage 1: Lightning β <-- Generates 5 Anchors in 400ms
β Skeleton Generator β
ββββββββββββββ¬βββββββββββββ
β
ββββββββββββββββ¬βββββββββββ΄ββββ¬βββββββββββββββ¬βββββββββββββββ
βΌ βΌ βΌ βΌ βΌ
[Worker 1] [Worker 2] [Worker 3] [Worker 4] [Worker 5]
(Expands #1) (Expands #2) (Expands #3) (Expands #4) (Expands #5)
β β β β β
ββββββββββββββββ΄βββββββββββ¬ββββ΄βββββββββββββββ΄βββββββββββββββ
β (Concurrent Parallel Return)
βΌ
βββββββββββββββββββββββββββ
β Stage 3: Assembler β <-- Final Blueprint assembled
βββββββββββββββββββββββββββ
Latency Benchmark: 2,500 Token Architectural Runbook
| Generation Paradigm | Time-to-First-Token (TTFT) | Total Generation Time | User Perception |
|---|---|---|---|
| Sequential Autoregressive | 1,200 ms | 38.4 seconds | Sluggish token stream |
| Skeleton-of-Thought (SoT) | 380 ms | 9.2 seconds (76% reduction) | Instantaneous outline, rapid parallel fill |
By splitting documents into an abstract skeletal frame followed by parallelized expansion, Skeleton-of-Thought delivers massive documents at the speed of lightning.