# Summary Schema

This document defines the structure and semantics of the `summary.json` files that contain AI-generated paper summaries.

## Overview

The `summary.json` file contains structured, AI-generated analysis of a paper. It is designed to:

- Provide consistent, machine-readable summaries
- Support research triage and discovery workflows
- Enable automated categorization and search
- Remain stable across different AI providers
- Use controlled vocabulary when available

## Schema Version 1.0

### File Structure

```json
{
  "schema_version": "1.0",
  "one_sentence_summary": "This paper introduces a novel neural architecture for...",
  "problem_statement": "Current approaches to X suffer from limitations...",
  "method_overview": "The authors propose a hybrid approach combining...",
  "main_results": "Experiments show 15% improvement over baselines...",
  "claimed_contributions": [
    "Novel attention mechanism design",
    "State-of-the-art results on ImageNet",
    "Theoretical analysis of convergence properties"
  ],
  "assumptions": [
    "Data is independently distributed",
    "Computational budget allows for large models"
  ],
  "limitations": [
    "Only evaluated on English text",
    "Requires significant computational resources",
    "Limited theoretical justification for design choices"
  ],
  "problem_tags": ["classification", "computer-vision", "optimization"],
  "technique_tags": ["neural-networks", "attention", "transformers"],
  "entities": ["ImageNet", "BERT", "ResNet", "CIFAR-10"],
  "relevance_to_user": 0.75,
  "recommended_sections": ["Section 3.2", "Algorithm 1", "Table 2"]
}
```

## Field Definitions

### Required Fields

#### `schema_version` (string)
- **Purpose**: Track format version for migration
- **Format**: Semantic version string (e.g., "1.0")
- **Required**: Yes

#### `one_sentence_summary` (string)
- **Purpose**: Concise paper overview for quick scanning
- **Guidelines**: 
  - One complete sentence, under 200 characters
  - Focus on the main contribution or finding
  - Avoid technical jargon when possible
- **Example**: "This paper introduces a new attention mechanism that improves transformer efficiency by 40% while maintaining accuracy."

### Core Content Fields

#### `problem_statement` (string)
- **Purpose**: What problem does this paper address?
- **Guidelines**:
  - 2-3 sentences maximum
  - Focus on the gap or limitation being addressed
  - Explain why this problem matters

#### `method_overview` (string)
- **Purpose**: High-level description of the approach
- **Guidelines**:
  - 3-4 sentences maximum
  - Focus on the key innovation or insight
  - Avoid detailed algorithmic descriptions

#### `main_results` (string)
- **Purpose**: Key empirical findings or theoretical results
- **Guidelines**:
  - Quantitative results when available
  - Highlight significance of improvements
  - Note any surprising or counterintuitive findings

### Structured Lists

#### `claimed_contributions` (array of strings)
- **Purpose**: Authors' stated contributions
- **Guidelines**:
  - Extract from paper's contribution list
  - Preserve authors' framing and claims
  - 3-6 items typically

#### `assumptions` (array of strings)
- **Purpose**: Key assumptions underlying the work
- **Guidelines**:
  - Mathematical, methodological, or data assumptions
  - Critical for understanding applicability
  - Often unstated but important

#### `limitations` (array of strings)
- **Purpose**: Acknowledged or apparent limitations
- **Guidelines**:
  - From authors' discussion or limitations section
  - Obvious limitations not acknowledged by authors
  - Important for understanding scope

### Categorization

#### `problem_tags` (array of strings)
- **Purpose**: Categorize the problem domain
- **Controlled vocabulary** (preferred values):
  - `classification`, `regression`, `clustering`
  - `optimization`, `search`, `planning`
  - `generation`, `translation`, `summarization`
  - `detection`, `segmentation`, `tracking`
  - `compression`, `encoding`, `decoding`
  - `privacy`, `security`, `robustness`
  - `interpretability`, `fairness`, `ethics`
  - `efficiency`, `scalability`, `deployment`

#### `technique_tags` (array of strings)  
- **Purpose**: Categorize the technical approaches
- **Controlled vocabulary** (preferred values):
  - `neural-networks`, `deep-learning`, `transformers`
  - `cnn`, `rnn`, `lstm`, `gru`, `attention`
  - `reinforcement-learning`, `supervised-learning`, `unsupervised-learning`
  - `bayesian`, `probabilistic`, `statistical`
  - `graph-neural-networks`, `graph-algorithms`
  - `computer-vision`, `natural-language-processing`
  - `federated-learning`, `transfer-learning`, `meta-learning`
  - `adversarial`, `generative-models`, `vae`, `gan`

### Entities and References

#### `entities` (array of strings)
- **Purpose**: Important datasets, models, algorithms, or systems mentioned
- **Guidelines**:
  - Proper names: "ImageNet", "BERT", "ResNet"
  - Algorithms: "SGD", "Adam", "RANSAC"  
  - Benchmarks: "GLUE", "COCO", "WMT"
  - Avoid generic terms like "neural network"

### User Relevance

#### `relevance_to_user` (number, optional)
- **Purpose**: Estimated relevance score for the user
- **Format**: Float between 0.0 and 1.0
- **Guidelines**:
  - Based on user's research interests (if known)
  - `null` if user preferences unavailable
  - Higher scores = more relevant

#### `recommended_sections` (array of strings, optional)
- **Purpose**: Specific sections worth reading in detail
- **Format**: Section references as they appear in paper
- **Examples**: ["Section 3.2", "Algorithm 1", "Table 2", "Appendix A"]

## Generation Guidelines

### AI Provider Instructions

When generating summaries, AI models should:

1. **Read for understanding**: Focus on the paper's core contributions
2. **Use structured thinking**: Work through each field systematically  
3. **Prefer facts over interpretation**: Extract what authors claim, not opinions
4. **Use controlled vocabulary**: Select from predefined tag lists when possible
5. **Be concise**: Optimize for quick scanning and search
6. **Handle uncertainty**: Use `null` or empty arrays for unclear fields

### Quality Criteria

Good summaries exhibit:
- **Accuracy**: Faithful to the paper's content
- **Completeness**: Cover all major aspects  
- **Consistency**: Similar papers get similar treatment
- **Searchability**: Use terms that aid discovery
- **Brevity**: Information density over verbosity

### Common Issues to Avoid

- **Hallucination**: Never invent facts not in the paper
- **Editorializing**: Don't add opinions about paper quality
- **Inconsistent terminology**: Use standard field names
- **Over-abstraction**: Keep concrete details when useful
- **Under-specification**: Provide enough detail for usefulness

## Schema Evolution

### Version History

- **v1.0** (current): Initial schema with core fields

### Migration Strategy

When the schema evolves:
1. New versions increment the `schema_version` field
2. Migration tools handle format upgrades automatically  
3. Backward compatibility maintained when possible
4. Deprecated fields are marked but preserved

### Extensibility

Future versions may add:
- Additional structured fields
- Hierarchical tag taxonomies
- Multi-lingual support
- Citation relationship mapping
- Experimental reproducibility metadata

## Integration with paperlib

### File Lifecycle

1. **Generation**: AI provider creates `summary.json`
2. **Validation**: paperlib validates against schema
3. **Indexing**: Content indexed for search
4. **Rendering**: Human-readable `summary.md` generated
5. **Updates**: Summaries can be regenerated with new models

### Search Integration

Summary fields are indexed for search:
- Full-text search includes all text fields
- Tag-based search uses `problem_tags` and `technique_tags`  
- Entity search uses the `entities` field
- Relevance ranking can use `relevance_to_user` scores

### API Integration

Higher-level tools can consume summaries programmatically:

```python
import json
from pathlib import Path

# Load summary
summary_path = Path("papers/arxiv/2022/arxiv-2212_06340/summary.json")
with summary_path.open() as f:
    summary = json.load(f)

# Extract key information  
tags = summary["problem_tags"] + summary["technique_tags"]
relevance = summary.get("relevance_to_user", 0.0)
entities = summary["entities"]
```

This enables automated workflows like:
- Daily digest generation
- Research recommendation systems  
- Literature review automation
- Cross-reference discovery

## Examples

### Machine Learning Paper
```json
{
  "schema_version": "1.0",
  "one_sentence_summary": "Introduces EfficientNet, a family of convolutional neural networks that achieve better accuracy and efficiency than previous models through compound scaling.",
  "problem_statement": "Existing ConvNet scaling methods arbitrarily scale network dimensions, leading to suboptimal accuracy and efficiency trade-offs.",
  "method_overview": "The paper proposes compound scaling that uniformly scales network width, depth, and resolution with a fixed ratio, guided by neural architecture search to find optimal scaling coefficients.",
  "main_results": "EfficientNet-B7 achieves 84.3% top-1 accuracy on ImageNet while being 8.4x smaller and 6.1x faster than the best existing ConvNet.",
  "claimed_contributions": [
    "Novel compound scaling method for ConvNets",
    "EfficientNet family with state-of-the-art accuracy/efficiency",
    "Systematic study of scaling dimensions"
  ],
  "assumptions": [
    "ImageNet classification transfers to other vision tasks",
    "Compound scaling works across different architectures"
  ],
  "limitations": [
    "Limited evaluation on tasks beyond image classification",
    "Scaling coefficients may not generalize to all architectures"
  ],
  "problem_tags": ["classification", "computer-vision", "efficiency"],
  "technique_tags": ["cnn", "neural-architecture-search", "model-scaling"],
  "entities": ["ImageNet", "MobileNet", "ResNet", "NASNet"],
  "relevance_to_user": null,
  "recommended_sections": ["Section 3.1", "Table 2", "Figure 2"]
}
```

This schema provides a foundation for consistent, structured paper analysis while remaining flexible enough to evolve with new research needs and AI capabilities.