我认为 RAM 分数 @xeophon 和我为 ATOM 工作制作的分数是一个非常可靠的指标。
我认为 RAM 分数 @xeophon 和我为 ATOM 工作制作的分数是一个非常可靠的指标。如果我只是根据我们的工件管理中 30 天的最高 RAM 分数进行排名，那么所有模型都非常可靠。 RAM 分数 @ t=（模型的累积下载量）@ t /（尺寸类别的前 10 个下载量）@ t RAM 实质上是查看模型在特定时间点（本例 t 为 30 天）的下载量，相对于该尺寸范围内所有时间段前 10 个模型的下载量。得分高于 1 意味着它有望成为该尺寸类别中下载量排名前 10 的模型。我认为对于 50B+ 型号最有意义。在 HuggingFace 上，单独的下载量通常并不好，但大型 MoE 在没有成为可靠模型的情况下获得大量采用的情况很少见。
查看英文原文
I think the RAM score @xeophon and I made for ATOM work is a really solid metric. If I just rank by the top RAM scores @ 30days within our artifacts curation, all the models are really solid. RAM score @ t= (model's cumulative downloads) @ t / (top-10 downloads for size class) @ t RAM is essentially looking at a model's downloads at specific points in time (this case t is 30 days), relative to the downloads of the top 10 models of all time in that size bracket. A score of above 1 means it's on track to be a top 10 downloaded model at that size category. Imo most meaningful for 50B+ models. Downloads alone normally isn't great on HuggingFace, but it's rare a large MoE gets a ton of adoption without being a solid model.
I think the RAM score @xeophon and I made for ATOM work is a really solid metric. If I just rank by the top RAM scores @ 30days within our artifacts curation, all the models are really solid. RAM score @ t= (model's cumulative downloads) @ t / (top-10 downloads for size class) @ t RAM is essentially looking at a model's downloads at specific points in time (this case t is 30 days), relative to the downloads of the top 10 models of all time in that size bracket. A score of above 1 means it's on track to be a top 10 downloaded model at that size category. Imo most meaningful for 50B+ models. Downloads alone normally isn't great on HuggingFace, but it's rare a large MoE gets a ton of adoption without being a solid model.
I think the RAM score @xeophon and I made for ATOM work is a really solid metric. If I just rank by the top RAM scores @ 30days within our artifacts curation, all the models are really solid. RAM score @ t= (model's cumulative downloads) @ t / (top-10 downloads for size class) @ t RAM is essentially looking at a model's downloads at specific points in time (this case t is 30 days), relative to the downloads of the top 10 models of all time in that size bracket. A score of above 1 means it's on track to be a top 10 downloaded model at that size category. Imo most meaningful for 50B+ models. Downloads alone normally isn't great on HuggingFace, but it's rare a large MoE gets a ton of adoption without being a solid model.
- 来源
- X @natolambert · x-natolambert
- 证据角色
- x_open_model_network
- 时间状态
- known
- 用户判决
- 未判