• Blue_Morpho@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    5 days ago

    Access patterns on neural nets are predictable long in advance. That means latency doesn’t matter much

    While the bag of experts approach has allowed segmentation of memory into vram and dram, the access in the particular expert isn’t predictable, access is spread across all the memory for that expert, and latency dramatically affects bandwith.