Realtime Onscreen Translator App with AI [greenfield]
The client is an industry-leading Asian development company. Their goal was well-defined: to create a real-time screen translation app that utilized AI technology to read all text on a user’s screen at all times, translating it from the original text to a user’s preferred language as instantly as possible. The client had a clear and concise idea about what they wanted to create and was simply in need of a development team who would turn their vision into a reality.
Objective
To build a cross-platform (MacOS, Windows) desktop app for onscreen translation. The goal is to keep all context of the original text as accurate as possible, which is achieved by drawing back all text to its original place on the screen, just in the user’s preferred language.
Process
1,The client planned the application from a business perspective.
2,Our teams worked together to plan the app from a technical perspective.
3,Our team developed several prototypes and iterated through designs and possible technologies until we reached a capable solution that ticked all the boxes.
4,We created the application and handed it over to the client for release.
5,We now provide ongoing support for new features and fixes.
6,We have a regular weekly meeting with the client where we discuss newly implemented features, upcoming feature requests, and bug fixes.
Scope
Desktop App Development
Microservice-based Backend Development
Tech stack
Typescript
MacOS
Windows
Python
NodeJS
OCR
Tensorflow
C++
Objective-C
Docker
Technical challenges
To find the adequate tech stack and AI models for our needs
To find ways to run AI models in the browser, if required
To create a performant, universal solution that is platform agnostic
To code missing modules and functionality in Native (Objective C++ for Mac and C++ for Windows)
To provide an application speed of around 1FPS or more for frequent updates when on-screen text changes or updates, such as side-scrolling text bars, live stream chats, etc.)
Outcome
We worked hard to ensure our team delivered the application on time while fulfilling all client expectations. We ensured we did, and the client was handed a fully functioning Desktop app (with the corresponding backend) by the agreed deadline. As we've developed a cross-platform app, we could really quickly iterate over new designs, features, change requests, and so on, saving both money and time for the client and allowing ourselves to carry out the work in the most efficient way possible. Moreover, most of the codebase is similar between platforms, so it's easier and cheaper to maintain. The only part of the codebase that varies between the Mac and Windows versions is the platform-specific logic that was inevitable to code in native.