Open-R1: a Totally Open Reproduction Of DeepSeek-R1

Hey there! This post is an intro to the task, not a claim that we have actually recreated R1 yet. We’re integrating in the open, so as quickly as we have evaluation numbers, we’ll share them. You can follow our progress on Hugging Face and GitHub.

True, but it seems like there’s absolutely nothing to be examined since today. I presume the supreme goal is to train a new reasoning design and after that use the very same assessment metrics as o1 and the DeepSeek-R1.

Well, there should be at least some peace of mind check and recognition to guarantee the design was trained correctly.

Oh yes, if you are discussing the examination number of deepseek’s design it’s coming really soon!

As mentioned in the post there is no design called Open-R1 to evaluate at all … not yet anyway. This is a blog site describing that Hugging face will take the R1 Deepseek design, work out how it was constructed as outlined in the paper and from what they launched, and then replicate that procedure.

in fact this is pretty much how science works … A develops a strategy, discovery or innovation and it is tested by B, C and D to see if it is reproduceable. Thats been the cornerstone of research now for a few centuries.

This blog site is not stating they have already done so … Its a blog site detailing an intent to begin training a design like R1 and calling it Open-R1.

Also DeepSeek-R1 was only released last week, and even in their paper they laid out the calculate hours needed. While those are low calculate hours for a SOTA model this does not indicate you can train said design in a week. I ‘d personally enjoy to be able to train a transformer design in a week, but we may require to wait a while for that level of compute technology.

So there are no standards for a design that has not been constructed yet right? As described in the blog site, and once again in reply to your question.

However fear not, there is a GitHub Repo currently and factors (hell I may join myself), some prelim work done, and a plan of attack. A great beginning position.

n
@edbeeching
has actually examined the released models currently

( src: https://x.com/edwardbeeching/status/1884273209136275742)

R1 just trained on o1 outputs, so jointly …/ s. This is what the new AI czars are stating

Hi! This article is an intro to the task, not a claim that we have actually replicated R1 yet. We will absolutely share the missing piece when we have them, you can expect the designs and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

That’s good and important to understand this remarkable hype that lacks technical comprehension and description. Science is about recreation, and if they declare to be open, let them fullfill the open part.

Please do publish the training expense.

We will!

Excalidraw Hi n
@bojan2501
thanks, we will certainly be striving to make certain this training recipe can work for small language designs on consumer hardware considering that not everyone has a cluster of H100s in your home:-RRB- The tool we utilized for the images was Excalidraw! https://excalidraw.com

eagerly anticipating it! WTF are your speaking about?

need to be a joke

It’s really cool to see how the entire open source community comes together!

Ops …

5.5 M is number reporter in the deepseekv3 tech report (simply the training, not the experiment afaik), for R1 tough to approximate tbh however much less than 5.5 M imo

Historically, they have never ever released code or datasets of their LLM training, so I would not expect this time to be various. If they would launch it that would be fantastic of course!

Yes naturally!

So basically you’re asking to change existing censorship with another flavour of censorship?

The code for the models are inside the model repositories, e.g. for V3: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py

Hello Team, I’m Ray Bernard, the author and creator of EQUATOR. My research group will be working on a paper concentrated on replicating certain components of DeepSeek R1. Our goal is to reproduce the cold start and provide your group with a dataset that consists of COT and other methods to support these efforts. We like to contribute our work to assist. Please let me understand if you find this beneficial. Best, Ray Bernard https://www.facebook.com/groups/1186310571520299/

Where is the evaluation numbers? without it you can’t call it recreation.

8 replies

True, however it appears like there’s nothing to be examined as of today. I assume the supreme goal is to train a brand-new thinking model and then utilize the same assessment metrics as o1 and the DeepSeek-R1.

That’s quite fascinating, I was asking myself why the questions the author exposed here are not being asked by others? I think the work they have done is remarkable however at the very same time I question why they would not put these missing pieces on if they are expected to be completely open.
Why even without reproduction and comprehension of the development they could affect so much the market in this way?

4 replies

Hi! This article is an introduction to the task, not a claim that we have actually reproduced R1 yet. We will absolutely share the missing out on piece when we have them, you can expect the designs and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

Interesting read, and it is good that we see more effort into this instructions: more optimization and less brute force.
Also wonder what tool did the author use for developing action diagram.

2 replies

Excalidraw I’m so pleased that initiative like this already exist, I’m gon na attempt to contribute:-RRB- 1 reply

looking forward to it! So racist articel

2 replies

WTF are your speaking about?

Awesome to have this open recreation began!

For Step # 1 check out https://github.com/open-thoughts/open-thoughts!

https://x.com/ryanmart3n/status/1884284101265612856

Let’s do this thing!

1 reply

It’s truly cool to see how the entire open source neighborhood comes together!

Does anyone know the actual training expense of r1? I can’t find it in the paper or the announcement post. Is the 6M expense reported by media simply the number drawn from v3’s training expense?

2 replies

Ops …

Has anyone asked the DeepSeek group to publish their training data and code, or a minimum of share them privately with an independent replication task like this? Have they declined such a request?

A loyal replication depends upon using the very same dataset and hyperparameters. Otherwise, any major disparities with the released criteria would be difficult to pin down-whether due to training information distinctions or the replication technique itself.

1 reply

Historically, they have actually never ever released code or datasets of their LLM training, so I would not expect this time to be various. If they would release it that would be fantastic naturally!

In the meantime we need to make finest guess estimates and see if we can arrive ourselves.

You provide great replication process of Deepseek reasoning training. I will attempt something to it.

This is really great information, can we fine tune with specific use case when code is launched?

1 reply

Yes obviously!

Please consider eliminating biased, polluted or unaligned training information and make an effort to eliminate copyrighted works from the crawl from intake. This will make the design more usable. If you reused anthropic curation checks, this might also assist, remove obviouslybiased information will likely add a lot of worth. We don’t desire another tainted, unaligned open source model, right? And no business would ever use deepseek or a design that recycles it, right?
We value your work for the benefit of mankind, we hope.
Miike C from NJ

1 reply

So basically you’re asking to change existing censorship with another flavour of censorship?

Can’t wait! Hopefully the model will be uncensored however whatever you can do is alright! Love seeing open source structure itself up. I’m not smart sufficient to actually help however I can contribute ethical support lol

Hello guys, I am even simply searching for code for DeepSeek-V2, in order to fully understand multi-head latent attention. You do not appear to have code in Hugging Face even for that. Or am I missing out on something? Don’t see anything in src/transformers/models. MLA is not appropriately explained in their paper, so it would be necessary to have code for this.