Balancing LLMs, Computer Vision, and Engineering Trade-offs

00:00:10: Hello, welcome to Dev Meo London week.

00:00:13: We are recording an episode of the signals podcast here live Live audience.

00:00:18: so thank you so much for being here.

00:00:20: I'm joined by Peter and Martin.

00:00:22: Thank You For Joining me Here Today.

00:00:25: So i guess we're going To Talk About Whatever you want to talk about but really kind Of The Initial Topic that i had in mind is?

00:00:33: We Are all Business Owners Entrepreneurs And involved in AI, and you two particularly are solving some really interesting challenges with AI that I don't know.

00:00:43: That could be done any other way.

00:00:45: So i would love to hear from each of you a little bit about kind Of what your problems?

00:00:49: You're solving right now

00:00:50: All right before it'll start.

00:00:52: so yeah i'm peter i'm the cto of Emma.

00:00:56: So emma is basically legal due diligence workspace.

00:00:59: so the idea Is whenever you sell or buy a company You want to be sure that all the contracts are in order, there's no legal liability there.

00:01:08: So typically during an M&A track or even investment you hire a law firm To read like thousands and thousands of documents... ...to find one needle-in-the-hay stack risk for a couple.

00:01:20: And what Emma does?

00:01:21: it basically automates this process In way.

00:01:25: lawyers can see every step if something off with the AI or can control it, or verify so that they don't have to read all of their documents anymore.

00:01:37: So that they just review whatever the AMMA workspace has found for them and save multiple days on a single deal.

00:01:51: Nice!

00:01:52: That sounds very exciting.

00:01:54: What do you got?

00:01:56: Basically I am Martin.

00:01:58: I own VMG, which is basically consulting firm for computer vision challenges.

00:02:03: We catered the B to B crowd so-to speak.

00:02:07: Basically if you are a software company that's providing your clients and have process they want automated then at one point computer vision might be very compelling option also very deterministic and compelling options.

00:02:25: As soon as you start the process to actually implement computer vision, You need to figure out a couple of things.

00:02:30: To make it stick in production and I think that's where most clients are calling us

00:02:36: Sure.

00:02:36: is there specific like a specific problem solving computer vision or just general?

00:02:42: Whatever your client wants to implement for their customers?

00:02:46: Yeah basically Most of the stuff the clients want to implement in computer vision space, but of course computer vision doesn't mean only RGB.

00:02:55: It means also can be satellite imaging for defense purposes... ...can be multi-camera system for sports tracking or can be MRI CT for medical applications.

00:03:05: But this was at the beginning a little difficult to explain.. ..but technology is always the same.

00:03:12: basically The recipes used are the same experience you bring to the table as they're all the same.

00:03:18: What modality and domain might change?

00:03:21: So that's why we cater to the BtoB-to-B market because basically, we try to approach software companies that want to automate something for their clients.

00:03:31: Beautiful!

00:03:34: I have a bunch of questions from both you.

00:03:35: there is so much to go with it There.

00:03:39: taking a moment talk about the legal aspect How are you solving the problem of kind of traceability?

00:03:48: You mentioned it.

00:03:49: The attorneys have to be able to see the process what that looks like.

00:03:52: That seems to be one of the things that's been challenging Recently, so what do you do and solve up?

00:03:57: yeah It is definitely challenging But what we do is obviously we find liabilities in contracts So the trace ability In our case is basically showing Like we found this liability And then just refer back To the right paragraphs in a document where the information comes from.

00:04:15: And that's one simple way of doing this.

00:04:18: So we always have to link back to the actual source so a lawyer can say like, okay That's what the contract says because there is zero trust.

00:04:28: cuz most lawyers they Play with chat GPT They write A bad prompt in general way too many questions.

00:04:37: wait you vague not enough context.

00:04:39: I mean I could.

00:04:40: I can talk for hours on all the things that they try which is basically the way I first did my own prompts, so i can't blame them.

00:04:49: But they get so bad results that they're like yeah chat gpt doesn't work for me where they generally get good results with drafting and there are many drafting startups out there.

00:05:02: but proper contract review really finding risks in contracts... That's pretty hard!

00:05:08: So one way of showing the document.

00:05:10: The other how we do traceability.

00:05:13: We show our customers what we check for.

00:05:16: So, we have a bunch of built-in checks legal things that we check and Our clients can add their own database of checks because every law firm often has like A specific set of things that they check for niche markets.

00:05:33: so we have Clients in renewable energy.

00:05:38: Regulations on windmill farms in Germany are different than they are in Denmark or Belgium, Or even the US.

00:05:44: Well especially recently.

00:05:47: So They will check for very different things and they will build In their own knowledge within the platform so that they have a Structured way of working which is something that they didn't have before.

00:06:02: usually you had to have just A bunch of juniors And if they're lucky they got a checklist.

00:06:06: I worked on it.

00:06:06: but That way, they can actually verify what's being checked for and what has the system found.

00:06:12: And why?

00:06:13: Is that really true or is it a case?

00:06:17: It's almost like maybe prompt engineering in getting good results as

00:06:22: hard!

00:06:23: But once you know the basics on... I think the best trick to provide all necessary context be very specific about your needs and limit the scope of what you're asking to a small question.

00:06:39: The problem is, if you limit this scope to a smaller question... You end up with thousands on questions over one thousand documents!

00:06:50: That's a million problems that we have to do easily in our case even more because we split it up further.

00:06:56: So it's not manageable to do that as a human, and that is basically the offering we do.

00:07:02: We say okay... ...we have cutted up in small pieces for you.

00:07:06: so I just want to check if this is what I call high-risk meaning risk low-risk.

00:07:11: You push button then go And everything is fully automated.

00:07:18: How does AI solve problems vastly different from your computer vision space?

00:07:29: The question sounds as biased as my answer probably.

00:07:35: I think there are many parts to an answer here but first of all, I think LLM's got incredibly popular because it is such a tangible product... ...I can actually pass my mobile phone through my parents with Gemini GPT or Claude and they can interact with that while I need like twenty minutes to explain what i'm doing with my parents, just to explain them actually.

00:08:02: You know we try and find there is something in this picture So it's not a sex.

00:08:07: It isn't as appealing.

00:08:08: And therefore also I think LLAMs got pushed through the boards Through CEO CTO into company While ML so-to speak.

00:08:18: computer vision Is now perceived old fashioned.

00:08:22: But In my opinion many ML use cases are underrated because you have such good measureability With all the clients that I've seen and have spoken, they are also interacting in the LLM space.

00:08:36: Benchmarking, validating... And especially bringing stuff to production is way more difficult than conventional use cases or not called it conventional because i think the landscape of computer vision shifted tremendously over the last couple years.

00:08:53: but we just have this scientific belief in metrics, numbers and objective acceptance criteria whereas LLMs I mean don't get me wrong how often do you vibe check your application?

00:09:06: Not enough.

00:09:07: that's the clear answer.

00:09:09: The honest answer

00:09:13: Yeah.

00:09:13: what i mean the term vibe checking is for me as a diehard ML enthusiast of last ten years already difficult one because tool I need to have results in terms of like.

00:09:28: is it ninety-six or ninety seven percent.

00:09:31: And i think this a huge paradigm shift, but also thing that's the biggest gap we have to bridge to bring LLMs truly into production ready maturity.

00:09:42: Yeah for sure!

00:09:43: Um...I

00:09:44: must say that um.. obviously made transition right from traditional machine learning even worked at computer vision company many years ago to nowadays LLMs, but what I really notice is that some things are just so easy now.

00:10:06: You can go so much faster and yes you have to validate quite a bunch of things with datasets But not everything really needs it.

00:10:15: there Are parts of the application?

00:10:17: That are not so key that you have To do this.

00:10:21: And yes, there are parts in our application where we do have to do this.

00:10:24: But you have some more leniency towards it which allows you go faster.

00:10:30: if I remember back into day all the companies that i worked for The amount of time We spent gathering data like training set test sets That part Like did the data engineering part was like Eighty to ninety percent of the time that we spent on a problem and then maybe there was like five or ten percent Of The Time left for the algorithms, And Then some Implementation On The Code Or Something Like That?

00:10:57: At That Part Is so Crucial.

00:10:58: but Like You Said For Computer Vision.

00:11:02: Like DD LLMs That Are Out There Today I wouldn't use them for computer vision.

00:11:06: you Can Ask an LLM is This A Fridge?

00:11:08: and yes you get a Reliable Answer but don't ask it.

00:11:11: much more specific things, like how many screws do you see in the image or something.

00:11:16: And then... It basically starts failing.

00:11:18: and these computer vision algorithms there I mean first breakthrough was in two thousand twelve.

00:11:26: They're still so immensely powerful Like that computer vision problem was in an interior solved around two thousand fifteen.

00:11:35: where we reached The first tasks were have human level performance Whereas language, it took us to twenty-twenty threeish before we started reaching human level performance in language.

00:11:49: So computer vision is just many years ahead even though you still need to use old techniques.

00:11:56: but yeah this better than I do.

00:11:58: there are ways how you can get by with less data and still get awesome results which supersede the human level of performance much better than what you can basically get with LLMs actually.

00:12:11: Absolutely, I think the good thing about the LLM space and development in the Llm spaces is that... You mentioned eighty to ninety percent of this data not still thinking it holds value up-to today but i just with a little less than eighty percent of the data, just because we have many techniques like self-supervised learning.

00:12:35: And there are really good and strong vision embeddings that you can leverage to bring your data requirements down to meaningful level on a POC level.

00:12:47: Of course as I said without going too much into tech ML methodology based because you build very pointy things that you can benchmark.

00:13:08: Yes, and this is I believe you nowadays engineers need to disfoundation And there's deep understanding To build pointy thinks that are benchmarkable be it LLM computers of the computer vision any space?

00:13:25: then You Can make projects stick Because if you rely only on prompting and reviewing it by hand than at one point you will probably...

00:13:34: And that's also the reason why, sorry for a technical term.

00:13:37: Why we use agentic workflows rather then agents?

00:13:42: The difference is in an agent work flow this steps are predefined.

00:13:46: You have all these steps on some of the steps to do with an LLM and you can scope That step has to be done very narrowly over optimize that one single prompt to get like really good results.

00:14:01: Whereas if you use an agent, and agent will determine the steps And every step of the way Figuring out what then the prompt for that step?

00:14:09: Will be the agent.

00:14:10: we'll come up with The answer and If he know how easily LLMs can hallucinate in come-up with things You know That's opening a Pandora box, and you see that at fifty percent of the time it works well.

00:14:23: And fifty per cent of the times get complete bullshit.

00:14:27: Companies like Cursor for example are mixing in together these fixed steps.

00:14:35: they provide as tools towards their agents so can control part of them.

00:14:41: So this mixed usage is off Agentech workflows combined with an agent that maybe picks which step to do in which situation.

00:14:51: And I think, That's the way how you engineer your way around this beyond what you have on a hype where can just say okay here is an agent and he can do everything but under the hood many of those steps are still agentech workflow.

00:15:06: for every step check How well does it perform?

00:15:09: Do

00:15:11: you mind

00:15:13: if I have one question because I find the angle very interesting?

00:15:17: Yeah,

00:15:17: sure.

00:15:20: You're talking about agentic workflows and fully-agentic...

00:15:23: Yep!

00:15:23: ...do you think once we solve benchmarking that FullyAgentic will unlock at next level

00:15:31: of quality?

00:15:33: Well, quality…I don't know but what you'll see is a continuation of the step where you have workflow steps that are fully controlled, but that the agent has just more tools to pick from.

00:15:54: I think that's the direction we're going into.

00:15:57: That is what i see with large platforms.

00:15:59: They won't tell us if it happens under the hood because its their agent who does everything and so fancy.

00:16:06: That's what I see happening, and that is the only way how you can guarantee some quality.

00:16:12: The tools they use are a simple API call or it could be using an LLM for a sub-task.

00:16:18: but we will uncover this next step with Open Claw, Hermit etc.

00:16:28: You end up providing more tools to these systems.

00:16:34: Using it even in Claude you can define these.

00:16:38: Yeah, I forgot the name where you start slash like this Skills thank-you.

00:16:46: luckily we have a good audience.

00:16:47: You can find these skills and basically that's what a skill is right?

00:16:52: It's you have like a prompt for specific sub problem And These LLMs can pick up these skills, and that's basically what your building.

00:17:01: you're building these agentic workflows in a way that an agent can just pick parts of it and assemble the weight once, you still get reliable output.

00:17:13: The way YOU want to see it!

00:17:18: Nice... So I've been using AI basically computer vision for about ten years now And ironically at this time as far as parking start-up was working with It started trying to assemble all these Photographs of license plates and vehicles were like, this is going to be a terrible training ordeal.

00:17:41: And somebody on the team was like hey look we should probably see if anybody else has already solved this problem?

00:17:47: Sure enough there's an API.

00:17:49: you could pay a few bucks per month then get these answers!

00:17:54: It's been fascinating for me to see almost commoditization in certain types processes and algorithms and data sets In the area space over a long period of time.

00:18:09: quite frankly I mean this was ten years ago when it was already well established.

00:18:13: in that space you would You pass in a photo or even a video Of vehicles, but he passed a photo vehicle and tells you to make them model The color license plate where its from like the country state of origin depending on the data.

00:18:27: It's fantastic And The biggest challenge we've had is actually convincing people that they want to give us a photograph of the entire vehicle and not just the license plate.

00:18:37: And it turns out in testing, you could take photo from the side of the vehicles as long as there's even a sliver on the license plates like ninety-five or ninety eight percent accurate ridiculous

00:18:49: How...

00:18:51: So this has been around for awhile.

00:18:52: how are things shifting now in the landscape, in terms of commoditization and these sorts things.

00:18:58: This is a situation where it's incredibly reliable input or output based on the inputs.

00:19:06: How are we going to see more commoditisation at this sort-of thing?

00:19:09: Is there gonna be more... The traditional approach like you're doing?

00:19:13: There's gotta be structured LLM approaches.

00:19:16: What do you think about

00:19:17: that?

00:19:18: I'll first admit big mistake.

00:19:21: as a speaker talking about many years ago, I was always saying like okay the people who own data will have all of their power.

00:19:31: Because within traditional machine learning that's what you needed to build and train your models.

00:19:36: in the moment when you owned this data You had these powers and could built it with LLMs completely changed And just showed how wrong i am back today because there is so much information on internet.

00:19:51: You even see that like in sales and marketing, people start publishing a lot of information online for free.

00:20:01: I think what you're doing with the podcast is very similar right?

00:20:06: You have some expertise but share your expertise through podcasts true talks so that people know and recognize you for your expertise.

00:20:16: then obviously they try to implement it might run into smaller issues.

00:20:20: Obviously, you're the most trusted person at that point to help out.

00:20:23: So even in sales and marketing You see that people are sharing lots of information.

00:20:28: so Information is becoming a commodity And That's thanks to LLMs that we're now able to process raw and dirty data In a way that doesn't require you say like okay this is them The input and that's the expected output.

00:20:47: No, you just give it a lot of text And go from there.

00:20:52: so I'm in that sense really positive In that sense that we see the commoditization Of information going forward Even large enterprises That have lots of proprietary Information.

00:21:06: You see smaller companies starting to share Because thats way they gain market share or the way They gained trust.

00:21:15: Podcast like this where you obviously will invite speakers that want to share what's happening under the hood rather than say, oh yeah but we have The Best Agent doing legal work.

00:21:25: No I'm not gonna tell You how it works right?

00:21:27: That's a very boring talk.

00:21:29: nobody Will listen To It.

00:21:30: so you see that that Commitization is not only a trend within data, but also in the entire industry.

00:21:38: So I see that actually progressing in the positive sense and it just shows again how wrong i was back in today!

00:21:52: I think like looking at traditional or computer vision space... ...I would not advise anybody to start up to the TechList place nowadays.

00:22:04: But what do you think?

00:22:07: That there is still value in training your own models.

00:22:10: There's still value

00:22:11: and

00:22:12: Controlling your own data because in the end you I think what is very underestimated?

00:22:19: In any of The Machine learning spaces that you are always fighting a try lemma so to speak You're finding trial emma of accuracy throughput, and latency And there are cases where you just need low latency for certain things.

00:22:37: An example is potato farming, for example.

00:22:41: You can detect potatoes that are not ripe with the camera system but basically it's a physical device where the potato drops in approximately less than fifty milliseconds to make decision and then you have to trigger an air blast that blasts the potato out of production line.

00:23:01: if you call an API endpoint, detect potato at whichever provider to choose by the moment that you get a response.

00:23:08: The potatoes have been probably already fried through French fries but this is a trial and I'm going off to solve them there.

00:23:15: in many cases as they'll believe it makes sense too be able to do with what your approach of problems vastly different than you did in two thousand eighteen because You can leverage student-teacher architectures, where you basically take a big model

00:23:33: that

00:23:34: supports humans in the loop to label your data or classify your data.

00:23:39: Or annotate and label their data And then basically... ...you take this knowledge preprepared and bake it into super small models That even works on an edge device like mobile phone So to speak.

00:23:55: And this is all feasible because we had these advancements in the last couple of years.

00:24:03: Yeah, but I even think just advocating for you that there are still many tasks which can't be solved with LLMs like especially computer vision.

00:24:15: Like I said, it will recognize a fridge but can't tell you how many screws there are in the picture or something like that.

00:24:20: Right?

00:24:20: You need custom models to get two reliable results.

00:24:24: and It's not only your speed and performance aspect.

00:24:28: That is something you basically get for free when you use Custom Models well For Free if you know what you're doing Obviously And then you Know How To Train An Efficient Model Anything that is not language-based, so we bridge the gap towards audio like spoken language.

00:24:52: But everything else... That's audio and video, image recognition even recommender systems Even predictive maintenance all these kind of things like anomaly detection signals those kinds time series prediction, those kind of things cannot be solved with LLMs because LLM's are as the name says large language models.

00:25:22: So you need to able define everything in language.

00:25:24: and yes for images we have data sets where they explain what is on that image.

00:25:28: it kinda helps graphs also working much better cause there lots of graphs online with people explaining how to read a graph.

00:25:37: So those kind of things work generally well, but just plain images?

00:25:43: That becomes... It's still very hard and will take us quite awhile when we'll catch up.

00:25:49: And that is something which is still quite far away.

00:25:53: Well, far A couple years out before you start seeing the breakthroughs And it won't solve the performance thing that custom models can offer.

00:26:03: you will never be able to roll them out at cost, right?

00:26:08: The trial and error throughput latency was basically an example.

00:26:11: but of course with going a custom path.

00:26:16: You can also influence the accuracy which is one thing.

00:26:22: in end I mean it boils down to the same thing as you do.

00:26:27: The point here, your spare is that you're trying to build in terms of which problem you are trying solve with a model... ...the more robust and accurate solution can be built.

00:26:36: And this holds true for computer vision things As probably for legal documents screening things Because the more precise or specific your task is Overfit on it.

00:26:55: Yeah,

00:26:55: and it goes much wider than just machine learning.

00:26:58: like if you have a business we Specifically chose within Emma to focus on one niche legal due diligence for M&A in investments.

00:27:08: There are large players out there Like Harvey and Ligora awesome firms with massive valuations.

00:27:14: We would never be able to compete them But they basically offer the Excel for lawyers right?

00:27:21: You can do Everything with them, but once you get to let's say use them two to track your customers.

00:27:28: Once you get too a hundred customers it becomes hard and you want to use a CRM.

00:27:33: We're more like the CRM.

00:27:35: But then for legal due diligence.

00:27:37: so It's not only within models though.

00:27:40: The more specific you are or the better the solution?

00:27:42: You can build its basically in everything.

00:27:45: So picking the right problem to solve and focus on that stays important.

00:27:52: As I discussed in my talk just earlier, even with LLMs nothing has changed.

00:28:00: We're still building the wrong things and solutions!

00:28:05: The more we can understand a problem that we are solving then AI gives us wings in many different ways whether it's machine learning or LLM or any other thing like writing code.

00:28:19: That is always key finding underlying problem to solve.

00:28:23: You heard it here folks, AI lets you build the wrong thing faster!

00:28:28: It's honestly what is happening...you see all over the place.

00:28:35: So one of those spaces that I spent a lot time in was web accessibility and so on are not yet solved problems.

00:28:50: good alt text for images.

00:28:52: And the reason it's not a solve problem is because there are two things required to make the alt text, we can describe images through reasonable level of accuracy.

00:29:00: today that's great.

00:29:01: unfortunately That does NOT make good Alt Text Good Alt Text also needs understand context of document or explanation and lesson in training happening that image support.

00:29:14: It need contextualize what is contained to the space that it's in.

00:29:21: And this is a clear situation where you kind of need a multi-pronged approach to truly solve this.

00:29:26: well, and I did some work with an education company recently, they had engineering diagrams... It turns out we needed models specifically trained on engineering diagrams.

00:29:48: I was impressed by how well those diagrams were explained, but then in order to truly contextualize this you kind of need your language model to be tailored to the subject matter at hand.

00:30:00: To do a really good job of explaining why is this diagram with all these circuits and whatever super relevant what i'm trying to teach you on material?

00:30:13: Those are for me interesting open spaces because it does require a wide range of approaches and skill sets, I think to solve them effectively.

00:30:22: What are your thoughts on something like that?

00:30:47: for legal documents.

00:30:49: And it took them until, I think two or three weeks ago to abandon that.

00:30:55: so For years they have only been fine-tuning their own model To stay ahead of the competition.

00:31:02: and then obviously Ligora comes along and we try as well Where we just pick off the shelf large language models?

00:31:10: We decided at Emma were not gonna fine tune because we noticed That The models are moving so fast that with prompt engineering we can get the same result.

00:31:19: In the beginning, we had to really engineer our prompts.

00:31:23: now it's even getting easier and I think for this situation you're describing is one of these niche cases where in two or three years You will still need fine-tune models.

00:31:40: realize this, you have two options.

00:31:43: You can either choose to say like okay I wait two or three years and i don't do anything about it?

00:31:48: And then... It becomes a commodity!

00:31:51: Then I launch but there will be fifty other companies doing the same.

00:31:55: Or you could choose that I'm going to use today with tools available for learning as much as possible but be aware of what new products and solutions are coming along, I make sure that they stay ahead.

00:32:11: There were very famous examples in the industry who did something similar... ...and now lead companies.

00:32:19: A good example is Spotify.

00:32:22: Back to my younger days when downloading stuff from the internet!

00:32:31: and didn't want to pay for it.

00:32:33: I was obviously into these peer-to-peer file sharing things, but I realized not so long ago that actually Spotify built like an entire peer-To-Peer File Sharing network if you click play on a song to let it start playing within the few seconds because their servers could not handle the load The internet did not support and they could have decided at that point, yeah we wait a couple of years until the servers can actually handle it.

00:33:07: And then we launch our product.

00:33:09: or we just built this peer-to-peer network right now today... ...and we spread out all the MPRGs over the world.. ..we work with peer to peer file sharing and get an instant response….

00:33:20: …that's what they did!

00:33:21: By doing so were their first ones – the frontrunners– to really capture the market To understand like what do people value when listening to music?

00:33:32: And that's where they're still the largest player in the world because They had this head start.

00:33:37: even now it has been fully commoditized and you have Google Music or whatever.

00:33:42: YouTube music Apple music, I don't know how many other players out there.

00:33:48: right.

00:33:49: so my view.

00:33:50: There is i think yes You Still need to find you but Don't let that be a reason to not start today because you build up a mode.

00:33:58: That will help You out in the future.

00:34:02: so I'm not convinced.

00:34:03: your wrong few years ago about The person who controls the data is just the data.

00:34:07: we're talking About his analytics, but people have the analytics of what People want how To serve it to them efficiently Own the marketplace.

00:34:16: a lot of cases and the opportunities for startups in many cases are The places where nobody owns that intelligence.

00:34:24: Yeah, yeah.

00:34:26: So there is still this this niches were.

00:34:28: the information is not publicly shared.

00:34:31: but even There I think gradually you will see that companies usually smaller players that want to disrupt and get attention, show off how smart they are.

00:34:44: Start sharing blog posts, podcasts... whatever on how to interpret those kind of data!

00:34:53: That just creates public knowledge like I think we were going past the stage really where knowledge ownership is what defines you as a company.

00:35:06: I think what really defines you is customer problem understanding, and that's the key.

00:35:12: That's the mode that you need to have as a company.

00:35:16: And the information?

00:35:17: That secondary.

00:35:18: many old school companies still hold on to it and these diagrams will be one of them but in time they will be disrupted just not yet today.

00:35:30: so for now You Have To Do It In A Different Way.

00:35:36: I agree in a certain way, because one and half years ago fine-tuning was the only way out or into an meaningful result.

00:35:51: Nowadays you can prompt yes but also see that if we don't start a journey then there is risk of being absorbed at some point by one of the big providers.

00:36:06: is there.

00:36:07: Because what part of value creation are you really providing to a customer?

00:36:13: Is it... The customers paying your money?

00:36:15: and basically just you're just wrapping their money into a prompt, then shipping them onto one of those big providers?

00:36:21: And I'm not sure if this will be enough down-the road.

00:36:27: Yeah!

00:36:28: I completely agree.

00:36:29: You see in legal space that Claude now has lots of skills for lawyers, which is basically what Harvey and Ligora offer.

00:36:39: Well they have one added thing that's tabular review.

00:36:43: but you can let Claude generate a table as well so you can put it in the skill to get there.

00:36:49: The table isn't just nice because its not proper user interface.

00:36:55: So you see that from the bottom up, Claude is eating away at Harvey and Ligura.

00:37:00: And probably they'll start finding niches like ours.

00:37:04: who knows where they started pushing the boundary... ...and we need to specialise and dig deeper.

00:37:10: so I completely agree with this part!

00:37:12: You have to start there today.. ..so you can stay ahead of the curve because otherwise it won't make it.

00:37:20: If you start today then basically collect data in a meaningful way and you get the chance to grow a market, collect data.

00:37:30: You have also fair chances of staying ahead in game.

00:37:33: but I just don't think that i would risk nowadays To start company as a rapper for one of big providers.

00:37:42: And this is not like against what your doing because...

00:37:47: In no way we are.

00:37:48: We're a rapper around LLMs But The Rapper more than Well a prompt that we offer.

00:37:57: We see the thing and thats what differentiates us alot from Harvey & Ligora, we build an entire user interface around legal due diligence workflow people have to do.

00:38:10: If you look at amount of time spent on prompt engineering And working on AI Thats like five or max ten percent product development time is spent on that user experience around.

00:38:27: That's where I think the difference will be, because sales force at a time they got feedback like you're just a wrapper round database which there were right?

00:38:41: It was like a user interface and everybody underestimated it.

00:38:45: but see what are today.

00:38:51: They built so much mode and logic on top of that, they are now so big.

00:38:57: nobody wants to use them anymore.

00:38:59: And people go into smaller products but still I mean...they're used everywhere!

00:39:04: Still today.

00:39:07: So i want back up just a little bit and piggyback with something you said earlier Martin about kind the approach.

00:39:14: You left out costs and curious how your approach varies when considering not only cost development But the cost of operation, how does that change your approach to what tools you're going use and build solutions?

00:39:35: That's a very good question.

00:39:39: Well I think in both spaces be it LLM or ML... ...I would just start with trying to achieve like best result again And as soon as i see that i get traction.. ..i will start to investigate how to cut costs, and we can do that in a meaningful way.

00:40:06: You maybe don't need Sonet or GPT-V for a simple intent classification at one point.

00:40:13: it's okay too use the smaller model.

00:40:15: even train your own classifier or something like that really bring cost down.

00:40:22: I think what most people underestimate is Once you deploy the model, that's a problem is solved.

00:40:31: And I think once we deploy it to model then real challenge starts because all these day two issues occur in production and they are not only cost-relevant but also things change.

00:40:47: We have seasonality.

00:40:49: Zalando doesn't need to sell winter clothes in peak summer One time during Covid three or four years ago where there was no toilet paper in Switzerland available.

00:41:01: These are all events that are basically obvious.

00:41:04: if you zoom out and look into the broader picture, I think once you ship something to production... You really have to start monitoring your costs for an understanding of how it might be interesting to invest money towards operational expenses toward upfront investment.

00:41:25: I'll also be honest and since this podcast is not going to be published online, that can be open about it.

00:41:31: Wait!

00:41:34: So okay just don't tell our investors but we like the solution that we offer.

00:41:40: We sell them below cost because The thing that we try to achieve at this moment Is showing the world?

00:41:48: That we can actually solve the problem.

00:41:52: We have a double gamble.

00:41:54: The double gamble is that on the one hand, LLMs get better and cheaper.

00:41:58: They they get better so that's clear.

00:42:01: Lately there not getting cheaper because of RAM shortages but at some point the cheaper models gets to this level.

00:42:08: then we can switch parts from our workflow into cheaper models.

00:42:12: And other thing is when you have enough traction it becomes reasonable to start optimizing Because I see many ways how we could optimize, but every optimization that we do makes us less flexible to adapt to new things that come along.

00:42:31: So for us it's like a double-edged sword where i basically say Like How much development cost It would be compared To the running costs That We have.

00:42:41: And yes, from the moment that the running cost becomes higher than development costs then we can optimize and reduce our cost.

00:42:48: I think if you do a five-fold cost decrease so they're fine but it will take some time and not enough load.

00:42:58: So... I think this is something many large organizations don't have the luxury of doing because when they launch something in production They need people like Martin who super optimise it and run at like a millisense per iteration, right?

00:43:19: Whereas in our case we just need to show that we can do before we optimise.

00:43:27: But I think this is basically the game where all the VC backed start-ups have to play Sell below costs.

00:43:36: I can't

00:43:36: tell it to my.

00:43:37: yeah,

00:43:37: of course If you look into for example code creation space i mean don't Tell me that the big the big players are Break even at all.

00:43:50: because i mean token prices Are basically going up with a better models currently and?

00:43:57: For long time i would have signed off That talk price will come down.

00:44:01: Because for me One of the promising things with Google is for example, they have basically a whole stack.

00:44:07: They own the stack down to the hardware and then basically also the intermediate layer models that ate everything.

00:44:14: so there will probably be ahead off The competition in terms of costs.

00:44:19: but with recent developments on demand exploding demand I'm just little worried.

00:44:26: Some startups will run out of funding because we might still see a wave of increasing token costs with the current hardware shortage.

00:44:34: Yeah, that can definitely happen.

00:44:36: That can happen.

00:44:38: What what?

00:44:39: We see?

00:44:40: Which makes it interesting is that the open weight models people like to call them open source but they're not The Open Weight Models.

00:44:48: They become better and better Like there was a talk.

00:44:51: I guy here in the conference John Davies.

00:44:54: He doesn't have a talk on the topic this time around, but he clearly shows that open weight models they're like half-a year behind.

00:45:03: On top tier models at least according to some benchmarks.

00:45:07: and yes hardware is still expensive.

00:45:10: But uh They are also innovating a lot on reducing running cost.

00:45:17: And there I think you will see more competition happening in weight models and other companies innovating way faster on reducing the running cost than that big players are doing.

00:45:30: Because it's still easier for us to just call an API endpoint, yes we have to do this in Europe so you're always lagging behind because of that reason.

00:45:40: Just make sure our data stays where its supposed to stay.

00:45:43: but at some point will also can switch through these open-weight models.

00:45:49: from what I see The running cost is at least half and if you would send your data to China, which obviously we can never do You can cut your costs by five.

00:46:00: If you Do it that way.

00:46:02: so there are ways where you see the market evolving in giving solutions.

00:46:06: but In general like every version of chat GPT since Version five up two five point five seems to be getting like slightly more expensive.

00:46:17: So were its not going in a good direction lately?

00:46:20: I see there another interesting statement in what you said.

00:46:26: You consider going to open-weight models, which is fine as soon as we start fine tuning open weight models?

00:46:35: What most people underestimate is suddenly the costs are on paper smaller but real costs might not even be smaller because all AI SRE topic behind it huge kind of firms for itself.

00:46:48: Then suddenly you don't only have a software reliability engineering team, but we also need an AISRE team and I think this burns more than just the two Xs as long as it can deploy Vanilla to one big

00:47:03: provider.

00:47:03: We will do that when we reach certain scale.

00:47:07: and there are these companies like Runpada.io Parts of the work for you, but still not everything because it adds a lot of complexity to your codebase cuz you need to start the pod and then well start with maybe Existing models.

00:47:25: And then until the pot is running and then start using that.

00:47:29: so It adds a load of complexity.

00:47:30: You need to manage and maintain that in becomes only relevant when you reach a certain scale at a certain

00:47:36: cost.

00:47:36: Yeah exactly now think this is this is something word.

00:47:40: small medium sized companies are struggling because They are forced to pay the twenty five or twenty dollars per million token on one of the big providers and they don't have that many alternatives, Because any maneuver they do with an upfront investment is so big That it takes them forever to actually save then on token prices.

00:48:03: Yeah I think If you want to succeed in this space and your scale into this size where this becomes relevant I think then you have a fair chance of surviving.

00:48:15: also the token price search.

00:48:18: Nice,

00:48:20: so we're basically at time.

00:48:21: but there's one more question that i really want to ask each of you and That is what is the biggest thing?

00:48:28: You perceive as a threat or an opportunity your Challenge that you're looking at for the near future like What something you kind of keeping your eye on in the future?

00:48:41: How broad or narrow do you want to define this?

00:48:43: Is it linked to our business, Or just society in general?

00:48:52: You could go either way.

00:48:54: I was thinking a little bit more nearly focused but like specific too Our roles as engineers and entrepreneurs.

00:49:03: Well specifically that regard everything is moving really fast Like things are progressing light speed if you follow it closely.

00:49:16: And, If your working at a large enterprise You still haven't noticed but I've been in startups since more than ten years... ...if i see how much shit can get done today How much faster everything is going?

00:49:33: How many more start-ups are pursuing the same thing?

00:49:38: It will be only a matter of time that major Large enterprises are getting disrupted and for me I'm in the startup space.

00:49:47: you might say well You're fine.

00:49:49: You won't worry because you're in the start-up space.

00:49:51: But even for us like we have to move really, really fast To stay ahead of the curve And to be able to compete with The Harvey's and Ligura is this world would their multi billion or even multi hundred billion dollar valuations with a small team, and being able to do that is really key.

00:50:12: That's why I've been reading about how Chinese companies are able to innovate so fast because there was the key there.

00:50:21: we're missing in Western world which i'm pursuing.

00:50:24: So it basically keeps me up at night.

00:50:30: apart from getting things to work The day-to-day struggle The slightly longer term, and for us a slightly longer time.

00:50:38: that's even the year right?

00:50:39: That is not super long-term because it so fast.

00:50:44: I think... ...the huge opportunity we have as We can do way more in the computer vision space than we could yesterday i think.

00:50:55: um the whole topic around data quality control.

00:51:01: everything came down to Ridiculously low prices because we can leverage so much more technology than what we could have done like Six seven years ago.

00:51:11: Even three years ago, it's just so much cheaper.

00:51:15: But I see also this whole AI thing as a threat Because i think AI is currently everywhere and I just think that um many people lose track of What?

00:51:27: Is there real business problem?

00:51:29: where is their business creating value?

00:51:31: which challenges are we trying to solve?

00:51:33: and That makes it for us at the moment a little more difficult to get into talks, because they see as we often get perceived old tech.

00:51:46: Whereas Dino VIII from Meta was released like two months ago and this is again game changer in computer vision space And I just hope somehow that will find reason of figuring out where you really want create value where companies make money and what the customers are willing to pay for.

00:52:09: And we're not chasing like every detour that is solvable with some sort of LLM automation?

00:52:16: Sure, great answer.

00:52:19: Thank you both very much.

00:52:20: thank You so much for being here.

00:52:22: This was a great conversation!

00:52:24: Thank you To who showed up in person live.

00:52:27: I appreciate you hanging out Again.

00:52:30: this is The Signals podcast.

00:52:31: Do check out DevMeo For all the latest and enjoy the rest of conference.

00:52:40: Thanks,

00:52:40: Mike!