Relevant source files

The following files were used as context for generating this wiki page: - [nanovllm/models/qwen3.py](https://github.com/geeeekexplorer/nano-vllm/blob/main/nanovllm/models/qwen3.py) - [nanovllm/engine/llm_engine.py](https://github.com/geeeekexplorer/nano-vllm/blob/main/nanovllm/engine/llm_engine.py) - [nanovllm/llm.py](https://github.com/geeeekexplorer/nano-vllm/blob/main/nanovllm/llm.py) - [nanovllm/__init__.py](https://github.com/geeeekexplorer/nano-vllm/blob/main/nanovllm/__init__.py) - [nanovllm/config.py](https://github.com/geeeekexplorer/nano-vllm/blob/main/nanovllm/config.py)

# LLM Class The `LLM` class in `nano-vllm` serves as the primary, high-level interface for users to interact with and run inference on large language models. It is designed to abstract away the complexities of model loading, tensor parallelism, request scheduling, and batched execution, providing a simple and intuitive API for generating text from prompts. This class is a direct subclass of the `LLMEngine` class, inheriting all its core functionalities. The `LLM` class itself acts as a convenient wrapper, exposing the essential methods needed for a typical user workflow, such as adding requests and generating completions. The underlying engine handles the intricate details of managing the model's lifecycle, processing requests efficiently, and coordinating with multiple processes if tensor parallelism is enabled. ## Class Hierarchy and Initialization The `LLM` class is part of a clear inheritance structure designed to separate user-facing logic from the core engine implementation. ```mermaid classDiagram class LLM { <> } class LLMEngine { <> -model_runner: ModelRunner -scheduler: Scheduler -tokenizer: AutoTokenizer -config: Config +__init__#40;model, **kwargs#41; +add_request#40;prompt, sampling_params#41; +step#40;#41; +generate#40;prompts, sampling_params#41; +is_finished#40;#41; } class Config { <> +model: str +max_num_batched_tokens: int +tensor_parallel_size: int +... } LLM --|> LLMEngine : inherits from LLMEngine --> Config : uses ``` As shown in the diagram, `LLM` inherits from `LLMEngine` ([`nanovllm/llm.py`](https://github.com/geeeekexplorer/nano-vllm/blob/main/nanovllm/llm.py)). The `LLMEngine`'s `__init__` method is responsible for setting up the entire inference environment. This process involves: 1. **Configuration**: It instantiates a `Config` object ([`nanovllm/config.py`](https://github.com/geeeekexplorer/nano-vllm/blob/main/nanovllm/config.py)) using the provided model path and other keyword arguments. This configuration holds all the necessary parameters for the engine, such as `max_num_seqs`, `gpu_memory_utilization`, and `tensor_parallel_size`. 2. **Tensor Parallelism**: If `config.tensor_parallel_size` is greater than 1, the engine spawns multiple worker processes using `multiprocessing` ([`nanovllm/engine/llm_engine.py`](https://github.com/geeeekexplorer/nano-vllm/blob/main/nanovllm/engine/llm_engine.py)). Each process runs a `ModelRunner` to handle a shard of the model. 3. **Component Initialization**: It initializes the main `ModelRunner` (for rank 0), a `Scheduler` to manage incoming requests, and an `AutoTokenizer` from the Hugging Face `transformers` library. The following flowchart illustrates the initialization sequence within the `LLMEngine` constructor. ```mermaid graph TB A[User instantiates LLM#40;model, ...#41;] --> B{LLM.__init__ calls LLMEngine.__init__}; B --> C[Create Config object]; C --> D{Check tensor_parallel_size #62; 1?}; D -- Yes --> E[Spawn worker processes for ModelRunner]; E --> F[Initialize main ModelRunner #40;rank 0#41;]; D -- No --> F; F --> G[Initialize Scheduler]; G --> H[Initialize AutoTokenizer]; H --> I[Engine is ready]; ``` ## Core Functionality and Workflow The primary use case for the `LLM` class is to generate text from one or more prompts. This is achieved through the `generate` method, which orchestrates a complete inference workflow from start to finish. ### The `generate` Method The `generate` method is a high-level abstraction that handles the entire lifecycle of a set of prompts. It takes a list of prompts (either as strings or token IDs) and corresponding `SamplingParams` for each. The workflow inside the `generate` method is as follows: 1. **Add Requests**: It iterates through the provided prompts and calls `add_request` for each one. The `add_request` method tokenizes the prompt (if it's a string) and creates a `Sequence` object, which is then added to the `Scheduler`'s queue. 2. **Inference Loop**: It enters a `while` loop that continues as long as there are unfinished sequences in the scheduler. In each iteration, it calls the `step` method. 3. **Step Execution**: The `step` method is the core execution unit. It asks the `Scheduler` for a batch of sequences to run, forwards this batch to the `ModelRunner` for inference, and then processes the returned token IDs, updating the state of each sequence. 4. **Collect Results**: Once a sequence is finished (e.g., it generates an EOS token), its final token IDs are collected. 5. **Decode and Return**: After the loop finishes, all collected token IDs are decoded back into strings using the tokenizer, and the final list of generated texts is returned. This process is visualized in the sequence diagram below. ```mermaid sequenceDiagram participant User participant LLM as LLM.generate() participant Scheduler participant ModelRunner participant Tokenizer User->>LLM: generate#40;prompts, params#41; loop for each prompt LLM->>Tokenizer: encode#40;prompt#41; Tokenizer-->>LLM: token_ids LLM->>Scheduler: add#40;Sequence#40;token_ids, params#41;#41; end loop while not finished LLM->>Scheduler: schedule#40;#41; Scheduler-->>LLM: batch_of_sequences LLM->>ModelRunner: run#40;batch#41; ModelRunner-->>LLM: output_token_ids LLM->>Scheduler: postprocess#40;batch, output_token_ids#41; end LLM->>Tokenizer: decode#40;final_token_ids#41; Tokenizer-->>LLM: generated_text LLM-->>User: #91;generated_text, ...#93; ``` ### Key Methods | Method | Description | Parameters | Returns | | :--- | :--- | :--- | :--- | | `__init__(model, **kwargs)` | Initializes the LLM engine, loading the model, tokenizer, and setting up parallel processes. | `model` (str): Path to the model directory.
`**kwargs`: Configuration overrides (e.g., `tensor_parallel_size`). | `LLM` instance | | `add_request(prompt, params)` | Adds a single generation request to the scheduler's queue. | `prompt` (str or list[int]): The input prompt.
`params` (SamplingParams): Parameters for sampling. | None | | `step()` | Executes one step of inference: schedules a batch, runs it, and updates sequence states. | None | `outputs` (list): Finished sequences.
`num_tokens` (int): Number of tokens processed. | | `generate(prompts, params)` | High-level method to generate completions for a list of prompts. | `prompts` (list): List of input prompts.
`params` (SamplingParams or list): Sampling parameters. | `list[dict]`: A list of dictionaries, each containing the generated `text` and `token_ids`. | | `is_finished()` | Checks if all scheduled requests have completed. | None | `bool`: `True` if finished, `False` otherwise. | ## Usage Example The `LLM` class is intended to be straightforward to use. It is exported directly from the `nanovllm` package for easy access ([`nanovllm/__init__.py`](https://github.com/geeeekexplorer/nano-vllm/blob/main/nanovllm/__init__.py)). ```python from nanovllm import LLM, SamplingParams # 1. Initialize the LLM engine # This will load the model, tokenizer, and start worker processes if needed. llm = LLM( model="/path/to/your/qwen3/model", tensor_parallel_size=2, # Use 2 GPUs max_num_seqs=256 ) # 2. Define the sampling parameters sampling_params = SamplingParams( temperature=0.8, top_p=0.95, max_tokens=150 ) # 3. Define the prompts prompts = [ "Hello, my name is", "The capital of France is", "In a world where", ] # 4. Generate the completions # The `generate` method handles the entire process. outputs = llm.generate(prompts, sampling_params) # 5. Print the results for i, output in enumerate(outputs): print(f"Prompt {i+1}: {prompts[i]}") print(f"Generated: {output['text']}\n") ``` ## Conclusion The `LLM` class provides a powerful yet simple interface for running large language model inference with `nano-vllm`. By inheriting from `LLMEngine`, it encapsulates a sophisticated system for parallel execution, memory management, and request scheduling. Users can leverage the `generate` method for a streamlined experience or interact with lower-level methods like `add_request` and `step` for more granular control over the inference process. This design makes `nano-vllm` accessible for both simple use cases and more complex, application-driven scenarios.