mistral_v0_2.model.rotary_embedding#
- mistral_v0_2.model.rotary_embedding.get_rotary_values_at_position(rotary_values, position)[source]#
Extracts the rotary positional embedding values for a specific position across all sequences in a batch.
- Parameters:
rotary_values (RotaryValues) – The rotary values from which to extract the positional embeddings.
position (Array) – The position for which to extract the rotary values.
- Returns:
Rotary embedding values for the specified position.
- Return type:
RotaryValues
- mistral_v0_2.model.rotary_embedding.make_rotary_values(batch_size, seq_len)[source]#
Generates sine and cosine values for rotary positional embeddings based on sequence length.
- Parameters:
batch_size (int) – The number of sequences in a batch.
seq_len (int) – The length of every sequences in a batch.
- Returns:
Rotary embedding values with sine values, and cosine values.
- Return type:
RotaryValues