Knowledge you can apply
Daily bookmarks: relevant news, facts and discoveries.
Radar: the post describes MOPD, a method in which a student model learns from its own rollouts and receives token-level supervision from domain-specialist teachers, aiming to combine the abilities of separately trained specialists.
Read articleRadar: a post claims EmbeddingGemma 2, from Google, maps text, code, images, video and audio into a single vector space and uses an Apache 2.0 license. The source gives no metrics or technical documentation.
Read articleRadar: the post presents JEPA-Anything, a domain-agnostic framework for world modeling based on orthogonal predictive factorization, which explores predictive representation learning across seven domains, from biology to climate.
Read articleRadar item: LiFT repeatedly applies a shared Diffusion Transformer core and, according to the publication, outperforms a dense DiT on ImageNet 256×256 with about 60% fewer parameters.
Read articleRadar: a report that 10 million EmbeddingGemma 2 vectors, quantized to 1 bit and reduced to 256 dimensions with MRL, dropped from about 30 GB to 0.4 GB of RAM, with a declared quality loss of about 5%.
Read articleBriefing on a paper presenting Native Hybrid Attention, an approach that seeks to balance the speed of Transformers in long contexts with the retention of precise details from linear attention.
Read articleRadar notes that Astral GPU Indexes now support PyTorch 2.13 and 2.14 and Python 3.15, with precompiled GPU-enabled wheels for FlashAttention, FlashAttention 3, causal-conv1d and DeepSpeed.
Read articleSummary of a paper that investigates attention sinks and residual sinks in large language models and proposes that these outliers, along with softmax attention and RMSNorm, rescale other components of the architecture.
Read articleRadar: the publication describes Dust, a zeroth-order optimization method that uses parallel perturbations of activations as a virtual population to pretrain transformers, with scale experiments and comparisons with backpropagation and evolution strategies.
Read articleRadar: an article on sharing memory across recursions in recurrent Transformers, reporting a 76–79% reduction in context memory for models from 150M to 1B parameters, compared with standard Transformers.
Read articleReport of scaling with nearly linear scalability and 35.3% MFU, using small-scale tests to narrow the search space before adding more GPUs.
Read articleRadar: the SwiLA paper, from COLM 2026, proposes switching between multiple linear maps and discusses the trade-off between attention expressiveness and KV cache memory cost.
Read articleBriefing on a section of the Machine Learning Engineering Open Book that describes symmetric memory in NCCL and PyTorch for small and medium payloads and for overlapping communication and computation.
Read articleRadar: according to a note dated October 4, 2026, Google released a 4B Gemma model, fine-tuned to correct speaker labels and reduce word-level diarization errors on four benchmarks.
Read articleA reported 450-million-parameter transformer trained with BF16 FlashAttention-3 saw its gradient norm rise 1,000× after 25 billion tokens, ending with higher loss than FP32 attention and no NaNs.
Read articlePaper proposes rewriting expert demonstrations as trajectories that are correct for the task and likely under the target model, before using them in standard SFT.
Read articleRadar briefing on a reinforcement learning setup for decision models that return calibrated probabilities instead of generated text. Publication date: October 4, 2026.
Read articleThe paper presents VeriHarness, which checks agent claims in long tasks using a fixed base model, without reference answers or rubrics during testing.
Read articleA systematic study reports that the direction of per-token KL matters more than the rollout policy in on-policy versus off-policy distillation, questioning the idea that on-policy rollouts are inherently better.
Read articleRadar: a paper that reads standard attention as Entropic Optimal Transport with an implicit uniform prior and proposes GOAT, with a continuous trainable prior that stays compatible with optimized kernels such as FlashAttention.
Read articleRota Nacional
30 days, no card, with a starting quota. After that, Pix credit from R$ 10,00.