Skip to content

vllm.v1.engine

Modules:

Classes:

CheckpointState

Bases: StrEnum

Process checkpoint lifecycle of an engine. It serves only while RUNNING, and may go through the cycle any number of times.

Attributes:

  • is_restored (bool) –

    Whether released state is back or coming back, so memory may wake and

Source code in vllm/v1/engine/__init__.py 
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
class CheckpointState(enum.StrEnum):
    """Process checkpoint lifecycle of an engine. It serves only while
    RUNNING, and may go through the cycle any number of times."""

    RUNNING = "running"
    # Draining, then releasing what a checkpoint must not contain.
    PREPARING = "preparing"
    # Safe to capture.
    PREPARED = "prepared"
    # Bringing back what preparing released.
    RESTORING = "restoring"
    FAILED = "failed"

    @property
    def is_restored(self) -> bool:
        """Whether released state is back or coming back, so memory may wake and
        scheduling resume."""
        return self in (CheckpointState.RUNNING, CheckpointState.RESTORING)

is_restored property

Whether released state is back or coming back, so memory may wake and scheduling resume.

EngineCoreEvent

Bases: Struct

A timestamped engine core event associated with a request.

The timestamp is a monotonic timestamp and is used by the engine frontend to calculate intervals between engine core events. These timestamps should not be compared with timestamps from other processes.

Source code in vllm/v1/engine/__init__.py 
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
class EngineCoreEvent(msgspec.Struct):
    """A timestamped engine core event associated with a request.

    The timestamp is a monotonic timestamp and is used by the engine
    frontend to calculate intervals between engine core events. These
    timestamps should not be compared with timestamps from other processes.
    """

    type: EngineCoreEventType
    timestamp: float

    @classmethod
    def new_event(
        cls, event_type: EngineCoreEventType, timestamp: float | None = None
    ) -> "EngineCoreEvent":
        timestamp = time.monotonic() if timestamp is None else timestamp
        return cls(event_type, timestamp)

EngineCoreEventType

Bases: IntEnum

The type of engine core request event.

Source code in vllm/v1/engine/__init__.py 
200
201
202
203
204
205
class EngineCoreEventType(enum.IntEnum):
    """The type of engine core request event."""

    QUEUED = 1
    SCHEDULED = 2
    PREEMPTED = 3

EngineCoreReadyResponse dataclass

Sent from EngineCore to each frontend at the end of engine startup.

Contains post-initialization config that may differ from the original values (e.g. max_model_len after KV cache auto-fitting).

Source code in vllm/v1/engine/__init__.py 
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
@dataclass
class EngineCoreReadyResponse:
    """Sent from EngineCore to each frontend at the end of engine startup.

    Contains post-initialization config that may differ from the original
    values (e.g. max_model_len after KV cache auto-fitting).
    """

    max_model_len: int
    num_gpu_blocks: int
    block_size: int
    dp_stats_address: str | None
    dtype: str
    vllm_version: str
    world_size: int
    data_parallel_size: int
    tensor_parallel_size: int
    pipeline_parallel_size: int
    decode_context_parallel_size: int
    data_parallel_rank: int
    max_num_seqs: int
    max_num_batched_tokens: int
    instance_id: str
    supports_lora: bool
    max_loras: int
    mamba_block_size: int | None = None
    # KV cache capacity (None for encoder-only/attention-free models).
    kv_cache_size_tokens: int | None = None
    kv_cache_max_concurrency: float | None = None
    kv_events_config: KVEventsConfig | None = None
    weight_transfer_backend: str | None = None
    enable_sleep_mode: bool = False
    supports_draft_weight_updates: bool = False
    # Full-attention block size in tokens after initialization, or unavailable.
    effective_attention_block_size: int | None = None
    coord_store_port: int = 0
    coordinator_input_address: str | None = None
    coordinator_output_address: str | None = None
    # JSON-mode dump of this engine's VllmConfig, for frontends that serve
    # `/server_info` without holding the config themselves (the Rust frontend).
    # Sent only with VLLM_SERVER_DEV_MODE, the same gate as `/server_info`.
    vllm_config: dict[str, Any] | None = None

EngineCoreRequest

Bases: Struct

Attributes:

Source code in vllm/v1/engine/__init__.py 
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
class EngineCoreRequest(
    msgspec.Struct,
    array_like=True,  # type: ignore[call-arg]
    omit_defaults=True,  # type: ignore[call-arg]
    gc=False,
):  # type: ignore[call-arg]
    request_id: str
    prompt_token_ids: list[int] | None
    mm_features: list[MultiModalFeatureSpec] | None
    sampling_params: SamplingParams | None
    pooling_params: PoolingParams | None
    arrival_time: float
    lora_request: LoRARequest | None
    cache_salt: str | None
    data_parallel_rank: int | None
    prompt_embeds: torch.Tensor | None = None

    # Per-position mask for mixed-mode inputs (e.g chat completion with
    # prompt_embeds content parts). `True` means the position is a real
    # token ID; `False` means the position uses a pre-computed entry from
    # `prompt_embeds`. `None` for pure-tokens and pure-embeds requests.
    prompt_is_token_ids: list[bool] | None = None

    # Index of the client, used to ensure outputs are sent back to the same
    # client for this request when scaling out the front-end.
    client_index: int = 0

    # Used in DP case to indicate which wave of requests this is expected to
    # belong to, to cover a race condition where the request is sent before
    # a wave finished notification is received.
    current_wave: int = 0
    priority: int = 0

    trace_headers: Mapping[str, str] | None = None
    resumable: bool = False

    # The user-provided request ID. This field is set internally,
    # copied from the provided request_id that's originally assigned
    # to the request_id field, see InputProcessor.assign_request_id().
    # Used in outputs and to support abort(req_id, internal=False).
    external_req_id: str | None = None

    reasoning_ended: bool | None = None
    reasoning_parser_kwargs: dict[str, Any] | None = None

    # If True, the request should be added to the scheduler's waiting queue
    # and immediately aborted, so connector-side cleanup runs via the standard
    # request_finished hook. Used to free P-side prefill blocks when a
    # KV-transfer request is rejected on the D node before engine admission.
    abort_immediately: bool = False

    session_id: str | None = None
    kv_hints: KvHintsEnvelope | None = None

    @property
    def params(self) -> SamplingParams | PoolingParams:
        """Return the processed params (sampling or pooling)."""
        if self.sampling_params is not None:
            return self.sampling_params
        assert self.pooling_params is not None
        return self.pooling_params

params property

Return the processed params (sampling or pooling).

EngineCoreRequestType

Bases: Enum

Request types defined as hex byte strings, so it can be sent over sockets without separate encoding step.

Source code in vllm/v1/engine/__init__.py 
324
325
326
327
328
329
330
331
332
333
334
335
336
class EngineCoreRequestType(enum.Enum):
    """Request types defined as hex byte strings, so it can be sent over sockets
    without separate encoding step.
    """

    ADD = b"\x00"
    ABORT = b"\x01"
    START_DP_WAVE = b"\x02"
    UTILITY = b"\x03"
    # Sentinel used within EngineCoreProc.
    EXECUTOR_FAILED = b"\x04"
    # Sentinel to wake up input_queue.get() during shutdown.
    WAKEUP = b"\x05"

FinishReason

Bases: IntEnum

Reason a request finished - stop, length, abort, error, or repetition.

Int rather than Str for more compact serialization.

stop - a stop string was emitted length - max_tokens was consumed, or max_model_len was reached abort - aborted by client error - retryable request-level internal error (e.g., KV load failure). Invariant: always converted to 500 Internal Server Error. repetition - repetitive token pattern detected (hallucination)

Source code in vllm/v1/engine/__init__.py 
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
class FinishReason(enum.IntEnum):
    """Reason a request finished - stop, length, abort, error, or repetition.

    Int rather than Str for more compact serialization.

    stop - a stop string was emitted
    length - max_tokens was consumed, or max_model_len was reached
    abort - aborted by client
    error - retryable request-level internal error (e.g., KV load failure).
            Invariant: always converted to 500 Internal Server Error.
    repetition - repetitive token pattern detected (hallucination)

    """

    STOP = 0
    LENGTH = 1
    ABORT = 2
    ERROR = 3
    REPETITION = 4

    def __str__(self):
        return FINISH_REASON_STRINGS[self.value]

ReconfigureRankType

Bases: IntEnum

Rank type for reconfiguring distributed request.

Source code in vllm/v1/engine/__init__.py 
349
350
351
352
353
class ReconfigureRankType(enum.IntEnum):
    """Rank type for reconfiguring distributed request."""

    KEEP_CURRENT_RANK = -1
    SHUTDOWN_CURRENT_RANK = -2