Seastarcharternautico

Overview

  • Founded Date March 16, 2022
  • Posted Jobs 0
  • Viewed 4

Company Description

Open-R1: a Totally Open Reproduction Of DeepSeek-R1

Hey there! This post is an intro to the job, not a claim that we have actually recreated R1 yet. We’re constructing in the open, so as soon as we have examination numbers, we’ll share them. You can follow our development on Hugging Face and GitHub.

True, but it looks like there’s absolutely nothing to be assessed as of today. I presume the ultimate goal is to train a brand-new reasoning model and then utilize the exact same assessment metrics as o1 and the DeepSeek-R1.

Well, there must be at least some peace of mind check and validation to ensure the model was trained correctly.

Oh yes, if you are talking about the examination variety of deepseek’s model it’s coming very soon!

As pointed out in the post there is no design called Open-R1 to evaluate at all … not yet anyhow. This is a blog detailing that Hugging face will take the R1 Deepseek design, exercise how it was as laid out in the paper and from what they launched, and after that reproduce that process.

in truth this is practically how science works … A creates a strategy, discovery or development and it is tested by B, C and D to see if it is reproduceable. Thats been the foundation of research now for a few centuries.

This blog site is not stating they have actually already done so … Its a blog site describing an intent to begin training a design like R1 and calling it Open-R1.

Also DeepSeek-R1 was just launched last week, and even in their paper they outlined the compute hours needed. While those are low calculate hours for a SOTA model this does not imply you can train said model in a week. I ‘d personally love to be able to train a transformer model in a week, but we may need to wait a while for that level of compute innovation.

So there are no criteria for a design that has not been built yet right? As described in the blog site, and once again in reply to your question.

However fear not, there is a GitHub Repo already and factors (hell I may join myself), some prelim work done, and a plan of attack. A great beginning position.

n
@edbeeching
has actually examined the released models currently

( src: https://x.com/edwardbeeching/status/1884273209136275742)

R1 just trained on o1 outputs, so jointly …/ s. This is what the brand-new AI czars are saying

Hi! This article is an introduction to the project, not a claim that we’ve recreated R1 yet. We will absolutely share the missing piece when we have them, you can anticipate the designs and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

That’s great and important to understand this incredible hype that does not have technical comprehension and description. Science has to do with reproduction, and if they claim to be open, let them fullfill the open part.

Please do release the training cost.

We will!

Excalidraw Hi n
@bojan2501
thanks, we will certainly be striving to make certain this training dish can work for small language models on customer hardware since not everyone has a cluster of H100s in the house:-RRB- The tool we utilized for the images was Excalidraw! https://excalidraw.com

looking forward to it! WTF are your discussing?

should be a joke

It’s actually cool to see how the entire open source neighborhood comes together!

Ops …

5.5 M is number reporter in the deepseekv3 tech report (simply the training, not the experiment afaik), for R1 hard to approximate tbh but much less than 5.5 M imo

Historically, they have actually never ever launched code or datasets of their LLM training, so I would not expect this time to be various. If they would release it that would be fantastic naturally!

Yes of course!

So basically you’re asking to replace existing censorship with another flavour of censorship?

The code for the designs are inside the design repositories, e.g. for V3: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py

Hello Team, I’m Ray Bernard, the author and creator of EQUATOR. My research group will be working on a paper focused on duplicating certain elements of DeepSeek R1. Our objective is to reproduce the cold start and supply your group with a dataset that includes COT and other strategies to support these efforts. We like to contribute our work to assist. Please let me know if you discover this beneficial. Best, Ray Bernard https://www.facebook.com/groups/1186310571520299/

Where is the examination numbers? without it you can’t call it recreation.

8 replies

True, but it appears like there’s nothing to be evaluated as of today. I presume the ultimate goal is to train a brand-new thinking design and then use the exact same assessment metrics as o1 and the DeepSeek-R1.

That’s rather fascinating, I was asking myself why the questions the author exposed here are not being asked by others? I think the work they have actually done is memorable however at the same time I wonder why they wouldn’t put these missing out on pieces on if they are supposed to be fully open.
Why even without recreation and comprehension of the development they could affect a lot the market in this way?

4 replies

Hi! This article is an introduction to the job, not a claim that we have actually replicated R1 yet. We will absolutely share the missing out on piece when we have them, you can anticipate the models and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

Interesting read, and it is great that we see more effort into this direction: more optimization and less strength.
Also wonder what tool did the author use for producing step diagram.

2 replies

Excalidraw I’m so thankful that initiative like this already exist, I’m gon na try to contribute:-RRB- 1 reply

anticipating it! So racist articel

2 replies

WTF are your discussing?

Awesome to have this open recreation started!

For Step # 1 check out https://github.com/open-thoughts/open-thoughts!

https://x.com/ryanmart3n/status/1884284101265612856

Let’s do this thing!

1 reply

It’s actually cool to see how the whole open source community comes together!

Does anybody understand the real training cost of r1? I can’t discover it in the paper or the announcement post. Is the 6M expense reported by media just the number drawn from v3’s training expense?

2 replies

Ops …

Has anyone asked the DeepSeek group to release their training data and code, or at least share them independently with an independent duplication task like this? Have they rejected such a request?

A loyal duplication depends upon utilizing the same dataset and hyperparameters. Otherwise, any major discrepancies with the released benchmarks would be hard to pin down-whether due to training data distinctions or the replication method itself.

1 reply

Historically, they have never launched code or datasets of their LLM training, so I wouldn’t expect this time to be different. If they would launch it that would be remarkable obviously!

In the meantime we need to make finest guess price quotes and see if we can arrive ourselves.

You provide good duplication procedure of Deepseek reasoning training. I will try something similar to it.

This is actually excellent details, can we fine tune with particular usage case when code is launched?

1 reply

Yes obviously!

Please consider eliminating prejudiced, polluted or unaligned training information and make an effort to remove copyrighted works from the crawl from consumption. This will make the model more usable. If you recycled anthropic curation checks, this may likewise help, eliminate obviouslybiased information will likely include a great deal of worth. We don’t want another polluted, unaligned open source design, right? And no corporate would ever use deepseek or a model that recycles it, right?
We value your work for the benefit of humankind, we hope.
Miike C from NJ

1 reply

So basically you’re asking to change existing censorship with another flavour of censorship?

Can’t wait! Hopefully the design will be uncensored but whatever you can do is alright! Love seeing open source structure itself up. I’m not clever sufficient to actually assist however I can contribute support lol

Hello guys, I am even simply searching for code for DeepSeek-V2, in order to completely understand multi-head hidden attention. You do not appear to have code in Hugging Face even for that. Or am I missing something? Don’t see anything in src/transformers/models. MLA is not properly explained in their paper, so it would be very important to have code for this.