Codesmith OSP code review: numerous "unbreak now" security vulnerabilities discovered after spending 5 minutes reviewing an "advanced security tool". Not the mid-level or senior engineering work it is claimed to be.
Codesmith OSP code review: numerous "unbreak now" security vulnerabilities discovered after spending 5 minutes reviewing an "advanced security tool". Not the mid-level or senior engineering work it is claimed to be.
I'm not going to share direct links because I don't want to pick on just this project or the people that made it. I circulated this post amongst a couple of Codesmith alumni to make sure they were ok with it as well.
What is the "OSP"? The OSP is the capstone project at Codesmith. You work in groups of 4-5 people, supervised by engineers. Codesmith claims it to be the key in making you a mid-level or senior engineer. It's the highlight of most alumni's resume and the main talking point in interviews.
I feel jerkish in posting about this widely instead of privately contacting the team that worked on it. But I've observed Codesmith's CEO, outcomes advisor, admissions staff, outcomes staff, social media posts, and alumni, all assure the public that Codesmith produces mid level and senior engineers capable of solving hard problems independently. I feel it is extremely important to balance that view.
**I'm also going to over-emphasize that 1. this is all my person opinions, on my own time, and 2. this is not a criticism as Codesmith as a whole or a "take down post" so if you support or don't support Codesmith, please don't pile onto this post. This is a post evaluating a sample of the engineering projects produced by Codesmith and I would encourage others to look into the OSLabs projects and do their own evaluations.**
**For a bootcamp project, I think this is a super cool idea and great 3-4 week long group project! I LOVE IT**. But if I'm applying my industry experience and judging it from the mid-level senior lens as the project is represented, I have concerns.
Context, This is an **advanced security tool** so I expected security to be considered seriously. I time-boxed the review to 5 minutes and 10 mins to write up this post, and another 10 mins editing it based on feedback from Codesmith alumni.
**This is my high level code review**:
1. The website doesn't have proper SSL setup. Many links in the Readme go to "example.com" or "insert your name here"
2. The .env file was checked in with ALL OF THE SECRETS AND KEYS for various 3rd party tools
3. Username and password for cloud services checked into the repo in plain text. A bad actor could destroy the demo DB or use it for nefarious purposes
4. Code has copied leftover files in it and WIP files that should be PRs and not checked in
5. Contains several cases of commented out code with no explanation
6. Authentication code console.logs important cookies for no reason, both a security issue and also bad practice to have personal developer debugging logging checked in.
7. No authenticationt/token check on a deletion endpoint, which could let a bad action take out the entire DB.
8. Several DB queries are doing inline string from user input so a bad actor could manipulate input to steal data or manipulate the database.
**Final note**, I read through random projects every so often and this was the only one I read today, maybe it's an edge case, but all of the marketing, Medium post, dozens of support comments about how good it is, GitHub starts, etc... would indicate it's a typical project. I see very similar things in projects frequently and have pointed them out privately before so while I don't think this is an edge case.
u/tputs001 wrote (the comment Michael replied to):
Lets be honest here, no bootcamp is outputting mid level to senior level engineers after only 3 months.
u/michaelnovatireplied·
I wouldn't post this if that list of people mentioned haven't adamantly insisted that in captured emails, recordings, slide decks and screenshots, but I strongly agree.
u/BudgetSense8077 wrote (the comment Michael replied to):
I liked this write up. Nice to see some of your programming chops. Leaving your sign in keys client facing is a big boo boo.
u/michaelnovatireplied·
As you can see on my GitHub, I spend most of the day coding and helping Fellows at Formation and surprisingly little time on Reddit haha.
https://github.com/mnovati
u/Sukuna_Kills_Gojo wrote (the comment Michael replied to):
Calm down it’s literally a project made by beginners who started a couple months ago. Ive seen working people accidentally commit credentials before, as long as the keys are trashed its not a big deal.
Michael no one actually believes codesmith is producing mid level engineers.
u/michaelnovatireplied·★ FEATURED
Well 3 people in the group say they worked on theos for 2 or 3 months on LinkedIn and it's listed as a Software Engineer job at a company so clearly people think this.
And I have a couple of emails from alumni to the effect of 'how dare you contact anyone in the Codesmith community about Formation, we don't need you and leave us alone, we are already mid level and senior engineers'
Of course people make mistakes and then they have to figure out how to fix that by rewriting the git history and changing all the credentials. In this case the credentials are all over the place and not just one bad commit.
The project is not close to any production code I've seen and is blatantly being portrayed as so.
u/NopeFish123 wrote (the comment Michael replied to):
I see some comments insisting a bootcamp is not enough for a job, but at least my bootcamp seemed to set realistic expectations. They said you should be trying predominantly for junior level roles and job search could take up to 6 months. They said to discuss projects as you woul
u/michaelnovatireplied·· edited★ FEATURED
Bootcamps are definitely enough to get a job and I'm also not debating that a minority of Codesmith grads get "senior" titles, but what I'm arguing is that the people are not actually mid level or senior engineers in both definition AND skill level.
I've worked with and given advice to a number of people in this position and tried to help them navigate their jobs - people just on Reddit who I don't even know their real names and it's really really really harmful to most people what Codesmith is marketing. Not "lawsuit level harmful" but like it's not the right career advice for most people there (even though it IS the right career advice for a minority of people there) and I have a deep passion for helping people have great CAREERS and not just the highest paying first jobs (where they will make way more money too across that great career than they will otherwise).
u/michaelnovati wrote (the comment Michael replied to):
As you can see on my GitHub, I spend most of the day coding and helping Fellows at Formation and surprisingly little time on Reddit haha.
https://github.com/mnovati
u/michaelnovatireplied·
I don't get the downvotes here, do people not believe I spend no time on Reddit or is this perceived as a humble brag? Engineers get better by writing code, not by posting and commenting on Reddit.
u/VastAmphibian wrote (the comment Michael replied to):
here's what I'd like to know regarding the resume massaging:
so you receive a resume. the applicant does not have a CS or STEM degree. applicant does not have a work history in the software field except for this one "company" that they've been at for 3 months. immediate questio
u/michaelnovatireplied·· edited★ FEATURED
I've interviewed a number of Codesmith grads for Formation acceptance (which is not a job, so I have a more constructive/feedback hat on and more tolerance) and they practice all of these questions at Codesmith yeah.
But yeah I noticed within 5 minutes, and the misleading answers kept going or we would have awkward silence, but people would not say it was a job, but they say it's something else. I was "working with an company under OS Labs" for example.
There are a number of buckets here but generally, this is why all of these jobs are with small or less well known companies - who are not tech companies, and don't have solid vetting processes, and sometimes people make it through.
1. People who get entry level jobs at solid tech companies that they call "mid level and senior" but aren't. e.g. someone at Google got entry level L3 job and said it was "level 3 senior" but L3 at Google is called "entry level" and the number 3 is an HR thing, not a seniority.
2. People who get mid level and senior jobs at non tech companies or at agencies or contractors. This is often where the "practice" and "messaging" works best to get past a generic recruiter screen. The companies are not super tech focused and people tend to get by. The roles themselves are often aren't for new grad/entry level engineers, but they are also "easier" and less intense then entry level FAANG roles. So I think it's fair to call these mid level and senior roles, but it doesn't mean the person who got them should be calling themselves a mid level and senior engineer. Or it's fine if they do but they don't portray themselves as "the outcomes of an elite graduate school" where people are getting entry level FAANG jobs paying much more. Like you get it one way or the other: mid level and senior jobs at okay-but-not-great companies, or you make amazing entry level engineers ready for the best jobs in the industry.... Codesmith is portraying that is prepares people for mid level and senior jobs AT the best companies in the industry.
3. People who get mid level and senior titles at startups. This is where it's fairly meaningless - the job postings were for senior roles but the companies needed competent engineers and the startup hired them for hustle and potential, but not "mid level and senior" skills.
4. People who lie. I've seen this flat out, "4 YOE" and believe it or not they get through the interviews. These people do sometimes get mid level and senior jobs at tech companies but it's quite the struggle. They can't ask for help or they will be "found out" and Codesmith doesn't have the experience to help them either. A number of these people change jobs quickly or are laid off, and some people just are really ambitious and figure out how to get by!
u/VastAmphibian wrote (the comment Michael replied to):
I work very closely with people in academia and this sounds just like the rounds of revisions a manuscript goes through. Not *exactly* forthcoming about the data, but technically not a flat out lie either. Many hours of meetings just to decide which exact word to use in a sentenc
u/michaelnovatireplied·★ FEATURED
Yeah, I've "written" two papers as an undergrad. One won a best paper award at a large conference... after the PhD students rewrote it in the "proper language" lol.
I think the difference is academia is heavily peer reviewed and collaborative and these projects have literally no one looking at the code.
But it's somewhat similar yeah
u/sheriffderek wrote (the comment Michael replied to):
It's a good thing most real websites aren't like this :P
Whatever you do... don't run the HTML through a validator
u/michaelnovatireplied·
Bad HTML that's invalid doesn't leak people's PII though :D
But yeah I actually hope generative AI can help with accessibility.
u/JayawardenepuraKotte wrote (the comment Michael replied to):
Michael, are there any bootcamps out there that prepare you skill wise for a mid/senior role? Or is it all just marketing? By bootcamp I mean in the traditional sense and not a career accelerator company like yours.
u/michaelnovatireplied·★ FEATURED
There are no bootcamps that I know of based on my definitions.
If you want high compensation, Codesmith, Launch School have well into six figure median salaries for placed students, and Rithm and Hack Reactor are close as well.
But there is no program that creates mid level and senior engineers because you can't get there without industry experience, but **let me explain what this means.**
I was promoted at Facebook from entry level to mid level in 3 months from starting and then mid level to senior in \~1.5 more years. The senior to staff in \~2 years.
So when I started, what was I? You could say 'well I was a mid level engineer from the start and underleveled!'
But that's really not true. I was an entry level engineer and I was treated like one, and I crushed it.
If I was hired as a mid level engineer, I might have underperformed or not done as well and maybe taken a lot longer to build the trust needed to get to senior. I had a fast trajectory but despite all the momentum in the world getting to mid level so fast, it took significant focus and work to get to senior.
So at the end of the day, you need that true on the job experience to develop the real world experience to level up and that can't be simulated.
What CAN BE TRAINED is if you HAVE THE EXPERIENCE BUT DON'T KNOW IT. So let's say you've worked for a year and having trouble leveling up. You might be able to reframe and reflect on that year in new ways and fill in gaps to get the most bang for your buck from that experience in the next job, or in getting promoted at your current job.
u/PsychologyIcy3577 wrote (the comment Michael replied to):
Cap. You've made 34 comments in the past 2 weeks alone. lmao
u/michaelnovatireplied·
**And I've made 218 GitHub contributions during that time**
Do your homework befor 'lmao' at me please.
u/PsychologyIcy3577 wrote (the comment Michael replied to):
such a fast reply for someone who's "not by posting and commenting on Reddit". It's ok bro, you're chronically online. me too haha
u/michaelnovatireplied·
Push notifications. I get a push, I spend literally under 2 mins replying and then I do 10 other things unrelated to Reddit, then I get a push again... you do you and I'll do me.
u/JayawardenepuraKotte wrote (the comment Michael replied to):
Are there further subdivisions to an entry level role? If these exist, I can picture how some bootcamps might want to target a not-so entry position.
u/michaelnovatireplied·★ FEATURED
Yeah, apprenticeships, internships, and and every company has completely different leveling systems.
I use [https://www.levels.fyi/](https://www.levels.fyi/) to compare the levels at different companies (don't get distracted by the salaries but just compare the granularity of levels)
You can see how Microsoft has more granular lower levels versus something like LinkedIn or Netflix.
u/DEV-Christopher wrote (the comment Michael replied to):
Yeah you’d think so, but that’s on the assumption that only bartenders are going through codesmith.
The dude that got a $400k+ offer this year from codesmith at a FAANG for example didn’t just skate on by with tricks. There’s a lot more nuance in this than simply an OSP or a re
u/michaelnovatireplied·· edited
What is that nuance? I've seen the entire curriculum and materials and I have my thoughts but I want to hear yours first.
u/annzilla wrote (the comment Michael replied to):
I had a codesmith grad contact me for a referral to my company. I didn't even submit it because I consider how they represented themselves on their resume as lying. This role was specifically for mid level and had a hard requirement of 2 years professional experience. I feel like
u/michaelnovatireplied·· edited★ FEATURED
Thanks for sharing. Yeah some people who push back on this think I have some kind of hidden interest in calling this behavior out, but it is solely because I know thousands of engineers, recruiters, product people that I've met during my Facebook days, who now work at hundreds of other companies, and the sentiment is unanimous that this behavior is anywhere from lying to fraud. Not one person has condoned if for any reason. That said, I very much understand the other point of view and acknowledge it as well: industry gatekeepers are blocking ambitious new engineers from getting a foot in the door so the ends justify the means. Which practical speaking, is a reasonable argument for the small number of people who genuinely fall in the bucket and then get appropriately levelled jobs they wouldn't get otherwise.
My concern is lack of transparency in behalf of Codesmith. When called out they double down: turning OSLabs into a non profit (that still has far too many ties to Codesmith internally when you go a layer deeper than what is public), or instead of correcting the guidance of how to properly add references to their industry sponsored tech talks, they defend their wrong guidelines, instead of telling people to make the OSLabs 3 week projects 3 weeks on a resume, they say they will sign a letter of reference for 3-4 months from OSLabs.
Like if the argument is the ends justify the means, tell it like it is! 100Devs (another free program, not Codesmith) tells it like it is in one of his Twitch live streams. Leon, their leader, explicitly tells people to lie on their resumes about working as a SWE for 100Devs to get past gatekeepers, hope it works eventually, and gives arguments for why.
I bet this comment will instantly get downvoted haha.
u/smallfrys wrote (the comment Michael replied to):
Is it any different than the marketing that the employers themselves do (e.g. great WLB, but my neighbors who work at G have to go in 3 d/week and always look exhausted despite being 10 years younger than me). Seems Codesmith is pushing people to get their highest value and not l
u/michaelnovatireplied·★ FEATURED
I agree that that's the view some people take and I understand where it comes from.
It's ultimately up to you to decide what you want to do. If you want to do that, go to Codesmith with the knowledge that this is how it works and be ready to go down that path.
My main goal here is to prevent people who DON'T WANT TO DO THAT from going to a program that has fantastic outcomes but isn't aligned in the how.
u/smallfrys wrote (the comment Michael replied to):
This makes sense, but are you saying all their grads are pushed to go after FAANG as mid/sr? Of just in general, and only the best performers can actually pass the gauntlet? Because that strikes me as a big difference.
I'd happily take the $125-150k after a bootcamp and get hir
u/michaelnovatireplied·★ FEATURED
So practically speaking, their grads are pushed to judge level via salary, e.g. if you get a 65K offer, you'll get a call from Eric convincing you not to take it, regardless of the actual position and if it's good for you long term, just based on the salary.
The thing they do is repeatedly tell people they are mid level and senior engineers just by finishing Codesmith. This sounds like there must be more to it, but the materials I've seen literally just start convos with, "Alright so since you graduated and are a mid level engineer, we'll have to do A, B, C on your resume" or "The OSP is the secret sauce that makes you a mid level or senior engineer by the end of Codesmith", or "Mid level engineers solve problems on their own and that's what Codesmith prepares you to do"
Yeah entry level FAANG is in the $150Ks base salary and about $200K with stock and bonuses, not including strong benefits and most importantly - impactful work that will open doors for the future.
u/Swami218 wrote (the comment Michael replied to):
It seems like there are execution issues in the project for sure. And obviously the execution is a huge part of it. I’d be interested to know what you think of the scope of these projects.
I suppose one could argue that even simple projects could be done in a ‘junior’ or ‘senio
u/michaelnovatireplied·★ FEATURED
It's really hard to answer this question. It's just impossible to work on this scope of project in 3 weeks. I would say that having a smaller scope project would LIMIT the potential, and the potential is NOT limited by the scope of the OSP, it's limited by the time.
Even if people worked 24 hours a day for two months it wouldn't be enough.
[Ada](https://adadevelopersacademy.org/) is a 11 month bootcamp, where everyone spends 6 months learning and then 5 months at a top tier internship with a partner company. It's a non profit with very strong partners like Zillow, Redfin, etc... And IF these people convert full time at the end of a 5 month full time internship, they are hired as entry level engineers.
Ada has paused enrollment because of the market and they can't guarantee those internships.
What makes Codesmith's 3 week OSP a secret sauce that makes it's alumni not just equal to a five month internship, but "mid level and senior" or "better" than a 5 month internship.